mlsanityData to 14 Oct 2026

Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation

arXiv 2506.05062 ↗First posted 5 Jun 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce Debate Speech Evaluation as a novel and challenging benchmark for assessing LLM judges. Evaluating debate speeches requires a deep understanding of the speech at multiple levels, including argument strength and relevance, the coherence and organization of the speech, the appropriateness of its style and tone, and so on. This task involves a unique set of cognitive abilities that previously received limited attention in systematic LLM benchmarking. To explore such skills, we leverage a dataset of over 600 meticulously annotated debate speeches and present the first in-depth analysis of how state-of-the-art LLMs compare to human judges on this task. Our findings reveal a nuanced picture: while larger models can approximate individual human judgments in some respects, they differ substantially in their overall judgment behavior. We also investigate the ability of frontier LLMs to generate persuasive, opinionated speeches, showing that models may perform at a human level on this task.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

24 of 41 references.

References · 24

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  3. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  6. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 154
  7. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  8. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  9. 2019A Large-scale Dataset for Argument Quality Ranking: Construction and Analysiscited by 67
  10. 2020From Arguments to Key Points: Towards Automatic Argument Summarizationcited by 61
  11. 2019Automatic Argument Quality Assessment -- New Datasets and Methodscited by 47
  12. 2019Corpus Wide Argument Mining -- a Working Solutioncited by 36
12 more
  1. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  2. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  3. 2020Quantitative Argument Summarization and Beyond: Cross-Domain Key Point Analysiscited by 32
  4. 2019Exploring the Role of Prior Beliefs for Argument Persuasioncited by 31
  5. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 29
  6. 2024Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLMcited by 21
  7. 2024Can Language Models Recognize Convincing Arguments?cited by 21
  8. 2024An Empirical Analysis on Large Language Models in Debate Evaluationcited by 20
  9. 2024Re-evaluating Automatic LLM System Ranking for Alignment with Human Preferencecited by 16
  10. 2020Out of the Echo Chamber: Detecting Countering Debate Speechescited by 15
  11. 2024DHP Benchmark: Are LLMs Good NLG Evaluators?cited by 12
  12. 2017A Recorded Debating Datasetcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Do LLM Debates Repeat Arguments Differently Across Languages?cited by 0
  2. 2026PolyDebate: A Game-Orchestrated Multimodal System for Debate Skills Practice and Evaluationcited by 0
  3. 2026Chinese Competitive Debating Dataset and Benchmarkcited by 0