mlsanityData to 7 Oct 2026

Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLM

arXiv 2403.08010 ↗First posted 12 Mar 2024cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

How can we construct an automated debate judge to evaluate an extensive, vibrant, multi-turn debate? This task is challenging, as judging a debate involves grappling with lengthy texts, intricate argument relationships, and multi-dimensional assessments. At the same time, current research mainly focuses on short dialogues, rarely touching upon the evaluation of an entire debate. In this paper, by leveraging Large Language Models (LLMs), we propose Debatrix, which makes the analysis and assessment of multi-turn debates more aligned with majority preferences. Specifically, Debatrix features a vertical, iterative chronological analysis and a horizontal, multi-dimensional evaluation collaboration. To align with real-world debate scenarios, we introduced the PanelBench benchmark, comparing our system's performance to actual debate outcomes. The findings indicate a notable enhancement over directly using LLMs for debate evaluation. Source code and benchmark data are available online at https://github.com/ljcleo/debatrix .

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 19 of its 37 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

19 of 37 references.

References · 19

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  5. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  6. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  7. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  10. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  11. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  12. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 83
7 more
  1. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  2. 2023Prompting Large Language Models With the Socratic Methodcited by 43
  3. 2023Exploring the Potential of Large Language Models in Computational Argumentationcited by 42
  4. 2019Are You Convinced? Choosing the More Convincing Evidence with a Siamese Networkcited by 29
  5. 2016Conversational flow in Oxford-style debatescited by 25
  6. 2022Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networkscited by 10
  7. 2021Machine Learning for Utility Prediction in Argument-Based Computational Persuasioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 6.

21 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  3. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMscited by 22
  4. 2025Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamicscited by 19
  5. 2024Can LLMs Beat Humans in Debating? A Dynamic Multi-agent Framework for Competitive Debatecited by 16
  6. 2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
  7. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  8. 2024Multi-Party Supervised Fine-tuning of Language Models for Multi-Party Dialogue Generationcited by 5
  9. 2025R-Debater: Retrieval-Augmented Debate Generation through Argumentative Memorycited by 4
  10. 2026RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mindcited by 4
  11. 2025Solving the Unsolvable: Translating Case Law in Hong Kongcited by 3
  12. 2025Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluationcited by 3
9 more
  1. 2025Strategic Planning and Rationalizing on Trees Make LLMs Better Debaterscited by 2
  2. 2025Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluationcited by 2
  3. 2026DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarioscited by 1
  4. 2024Overview of the CAIL 2023 Argument Mining Trackcited by 0
  5. 2025TurQUaz at CheckThat! 2025: Debating Large Language Models for Scientific Web Discourse Detectioncited by 0
  6. 2026Towards Robust Argumentative Essay Understanding via TIDE: An Interactive Framework with Trial and Debatecited by 0
  7. 2026Do LLM Debates Repeat Arguments Differently Across Languages?cited by 0
  8. 2026PolyDebate: A Game-Orchestrated Multimodal System for Debate Skills Practice and Evaluationcited by 0
  9. 2026Chinese Competitive Debating Dataset and Benchmarkcited by 0