# An Empirical Analysis on Large Language Models in Debate Evaluation

arXiv: https://arxiv.org/abs/2406.00050 · HTML: https://mlsanity.com/paper/2406.00050/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs](https://mlsanity.com/paper/2401.02582/) (2024)
- [VideoXum: Cross-modal Visual and Textural Summarization of Videos](https://mlsanity.com/paper/2303.12060/) (2023)
- [Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems](https://mlsanity.com/paper/2310.05280/) (2023)
- [SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluation](https://mlsanity.com/paper/2310.00074/) (2023)
- [Exploring the Role of Argument Structure in Online Debate Persuasion](https://mlsanity.com/paper/2010.03538/) (2020)
- [A Corpus for Modeling User and Language Effects in Argumentation on Online Debating](https://mlsanity.com/paper/1906.11310/) (2019)
- [Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networks](https://mlsanity.com/paper/2203.14647/) (2022)
- [LLMVA-GEBC: Large Language Model with Video Adapter for Generic Event Boundary Captioning](https://mlsanity.com/paper/2306.10354/) (2023)

## Built on by (most cited first, 20 of 20)

- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [Evaluating the Performance of Large Language Models via Debates](https://mlsanity.com/paper/2406.11044/) (2024)
- [VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?](https://mlsanity.com/paper/2411.10979/) (2024)
- [Kubrick: Multimodal Agent Collaborations for Synthetic Video Generation](https://mlsanity.com/paper/2408.10453/) (2024)
- [Training Language Models to Critique With Multi-agent Feedback](https://mlsanity.com/paper/2410.15287/) (2024)
- [Bias in the Mirror: Are LLMs opinions robust to their own adversarial attacks ?](https://mlsanity.com/paper/2410.13517/) (2024)
- [Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks](https://mlsanity.com/paper/2507.17747/) (2025)
- [MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness](https://mlsanity.com/paper/2505.20426/) (2025)
- [Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation](https://mlsanity.com/paper/2506.05062/) (2025)
- [Diffusion-Based Image Editing: An Unforeseen Adversary to Robust Invisible Watermarks](https://mlsanity.com/paper/2511.05598/) (2025)
- [Bias-Adjusted LLM Agents for Human-Like Decision-Making via Behavioral Economics](https://mlsanity.com/paper/2508.18600/) (2025)
- [The Role of Model Confidence on Bias Effects in Measured Uncertainties for Vision-Language Models](https://mlsanity.com/paper/2506.16724/) (2025)
- [DS@GT at Touch'e: Large Language Models for Retrieval-Augmented Debate](https://mlsanity.com/paper/2507.09090/) (2025)
- [CRAwDAD: Causal Reasoning Augmentation with Dual-Agent Debate](https://mlsanity.com/paper/2511.22854/) (2025)
- [Debate with Images: Detecting Deceptive Behaviors in Multimodal Large Language Models](https://mlsanity.com/paper/2512.00349/) (2025)
- [DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios](https://mlsanity.com/paper/2606.07226/) (2026)
- [FADE: Adversarial Concept Erasure in Flow Models](https://mlsanity.com/paper/2507.12283/) (2025)
- [Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustness](https://mlsanity.com/paper/2509.12024/) (2025)
- [MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluation](https://mlsanity.com/paper/2602.09624/) (2026)
- [DS@GT ARC at Touch'e: Large Language Models for Retrieval-Augmented Debate](https://mlsanity.com/paper/2608.08143/) (2026)

## Abstract (arXiv)

In this study, we investigate the capabilities and inherent biases of advanced large language models (LLMs) such as GPT-3.5 and GPT-4 in the context of debate evaluation. We discover that LLM's performance exceeds humans and surpasses the performance of state-of-the-art methods fine-tuned on extensive datasets in debate evaluation. We additionally explore and analyze biases present in LLMs, including positional bias, lexical bias, order bias, which may affect their evaluative judgments. Our findings reveal a consistent bias in both GPT-3.5 and GPT-4 towards the second candidate response presented, attributed to prompt design. We also uncover lexical biases in both GPT-3.5 and GPT-4, especially when label sets carry connotations such as numerical or sequential, highlighting the critical need for careful label verbalizer selection in prompt design. Additionally, our analysis indicates a tendency of both models to favor the debate's concluding side as the winner, suggesting an end-of-discussion bias.