mlsanityData to 7 Oct 2026

Evaluating the Performance of Large Language Models via Debates

arXiv 2406.11044 ↗First posted 16 Jun 2024cs.CL cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are rapidly evolving and impacting various fields, necessitating the development of effective methods to evaluate and compare their performance. Most current approaches for performance evaluation are either based on fixed, domain-specific questions that lack the flexibility required in many real-world applications, or rely on human input, making them unscalable. To address these issues, we propose an automated benchmarking framework based on debates between LLMs, judged by another LLM. This method assesses not only domain knowledge, but also skills such as argumentative reasoning and inconsistency recognition. We evaluate the performance of various state-of-the-art LLMs using the debate framework and achieve rankings that align closely with popular rankings based on human input, eliminating the need for costly human crowdsourcing.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 21 of its 43 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

21 of 43 references.

References · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  10. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
9 more
  1. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  2. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  3. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  4. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  5. 2023Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debatecited by 94
  6. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  7. 2023Let Models Speak Ciphers: Multiagent Debate through Embeddingscited by 49
  8. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debatecited by 40
  9. 2024An Empirical Analysis on Large Language Models in Debate Evaluationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 3.

19 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  4. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  5. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  6. 2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
  7. 2025Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysiscited by 16
  8. 2025Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarkscited by 7
  9. 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
  10. 2025Efficient MAP Estimation of LLM Judgment Performance with Prior Transfercited by 4
  11. 2025Statistical Methods in Generative AIcited by 4
  12. 2025AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefscited by 4
7 more
  1. 2026Multi-Agent Debate Strategies: Survey, Taxonomy, and Challengescited by 3
  2. 2025Self-evolving expertise in complex non-verifiable subject domains: dialogue as implicit meta-RLcited by 1
  3. 2025ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speechcited by 1
  4. 2025Concentration bounds on response-based vector embeddings of black-box generative modelscited by 1
  5. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  6. 2026Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Gamecited by 0
  7. 2026ToxiTrace: Gradient-Aligned Training for Explainable Chinese Toxicity Detectioncited by 0