# Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLM

arXiv: https://arxiv.org/abs/2403.08010 · HTML: https://mlsanity.com/paper/2403.08010/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations](https://mlsanity.com/paper/2307.02762/) (2023)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons](https://mlsanity.com/paper/1909.03087/) (2019)
- [Large Language Models are Diverse Role-Players for Summarization Evaluation](https://mlsanity.com/paper/2303.15078/) (2023)
- [Prompting Large Language Models With the Socratic Method](https://mlsanity.com/paper/2303.08769/) (2023)
- [Exploring the Potential of Large Language Models in Computational Argumentation](https://mlsanity.com/paper/2311.09022/) (2023)
- [Are You Convinced? Choosing the More Convincing Evidence with a Siamese Network](https://mlsanity.com/paper/1907.08971/) (2019)
- [Conversational flow in Oxford-style debates](https://mlsanity.com/paper/1604.03114/) (2016)
- [Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networks](https://mlsanity.com/paper/2203.14647/) (2022)
- [Machine Learning for Utility Prediction in Argument-Based Computational Persuasion](https://mlsanity.com/paper/2112.04953/) (2021)

## Built on by (most cited first, 21 of 21)

- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs](https://mlsanity.com/paper/2404.10160/) (2024)
- [Synthetic Socratic Debates: Examining Persona Effects on Moral Decision and Persuasion Dynamics](https://mlsanity.com/paper/2506.12657/) (2025)
- [Can LLMs Beat Humans in Debating? A Dynamic Multi-agent Framework for Competitive Debate](https://mlsanity.com/paper/2408.04472/) (2024)
- [Training Language Models to Win Debates with Self-Play Improves Judge Accuracy](https://mlsanity.com/paper/2409.16636/) (2024)
- [A Survey of Scaling in Large Language Model Reasoning](https://mlsanity.com/paper/2504.02181/) (2025)
- [Multi-Party Supervised Fine-tuning of Language Models for Multi-Party Dialogue Generation](https://mlsanity.com/paper/2412.05342/) (2024)
- [RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mind](https://mlsanity.com/paper/2601.15715/) (2026)
- [R-Debater: Retrieval-Augmented Debate Generation through Argumentative Memory](https://mlsanity.com/paper/2512.24684/) (2025)
- [Solving the Unsolvable: Translating Case Law in Hong Kong](https://mlsanity.com/paper/2501.09444/) (2025)
- [Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation](https://mlsanity.com/paper/2506.05062/) (2025)
- [Strategic Planning and Rationalizing on Trees Make LLMs Better Debaters](https://mlsanity.com/paper/2505.14886/) (2025)
- [Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation](https://mlsanity.com/paper/2509.19880/) (2025)
- [DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios](https://mlsanity.com/paper/2606.07226/) (2026)
- [Overview of the CAIL 2023 Argument Mining Track](https://mlsanity.com/paper/2406.14503/) (2024)
- [TurQUaz at CheckThat! 2025: Debating Large Language Models for Scientific Web Discourse Detection](https://mlsanity.com/paper/2508.08265/) (2025)
- [Towards Robust Argumentative Essay Understanding via TIDE: An Interactive Framework with Trial and Debate](https://mlsanity.com/paper/2605.17247/) (2026)
- [Do LLM Debates Repeat Arguments Differently Across Languages?](https://mlsanity.com/paper/2607.23442/) (2026)
- [PolyDebate: A Game-Orchestrated Multimodal System for Debate Skills Practice and Evaluation](https://mlsanity.com/paper/2608.16276/) (2026)
- [Chinese Competitive Debating Dataset and Benchmark](https://mlsanity.com/paper/2609.21637/) (2026)

## Abstract (arXiv)

How can we construct an automated debate judge to evaluate an extensive, vibrant, multi-turn debate? This task is challenging, as judging a debate involves grappling with lengthy texts, intricate argument relationships, and multi-dimensional assessments. At the same time, current research mainly focuses on short dialogues, rarely touching upon the evaluation of an entire debate. In this paper, by leveraging Large Language Models (LLMs), we propose Debatrix, which makes the analysis and assessment of multi-turn debates more aligned with majority preferences. Specifically, Debatrix features a vertical, iterative chronological analysis and a horizontal, multi-dimensional evaluation collaboration. To align with real-world debate scenarios, we introduced the PanelBench benchmark, comparing our system's performance to actual debate outcomes. The findings indicate a notable enhancement over directly using LLMs for debate evaluation. Source code and benchmark data are available online at https://github.com/ljcleo/debatrix .