mlsanityData to 7 Oct 2026

An Empirical Analysis on Large Language Models in Debate Evaluation

arXiv 2406.00050 ↗First posted 28 May 2024cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

In this study, we investigate the capabilities and inherent biases of advanced large language models (LLMs) such as GPT-3.5 and GPT-4 in the context of debate evaluation. We discover that LLM's performance exceeds humans and surpasses the performance of state-of-the-art methods fine-tuned on extensive datasets in debate evaluation. We additionally explore and analyze biases present in LLMs, including positional bias, lexical bias, order bias, which may affect their evaluative judgments. Our findings reveal a consistent bias in both GPT-3.5 and GPT-4 towards the second candidate response presented, attributed to prompt design. We also uncover lexical biases in both GPT-3.5 and GPT-4, especially when label sets carry connotations such as numerical or sequential, highlighting the critical need for careful label verbalizer selection in prompt design. Additionally, our analysis indicates a tendency of both models to favor the debate's concluding side as the winner, suggesting an end-of-discussion bias.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

16 of 33 references.

References · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  5. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  6. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  7. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  8. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  9. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  10. 2023VideoXum: Cross-modal Visual and Textural Summarization of Videoscited by 41
  11. 2023Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systemscited by 37
  12. 2023SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluationcited by 27
4 more
  1. 2020Exploring the Role of Argument Structure in Online Debate Persuasioncited by 26
  2. 2019A Corpus for Modeling User and Language Effects in Argumentation on Online Debatingcited by 22
  3. 2022Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networkscited by 10
  4. 2023LLMVA-GEBC: Large Language Model with Video Adapter for Generic Event Boundary Captioningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 3.

20 papers.

Sort:

By area:

  1. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  2. 2024Evaluating the Performance of Large Language Models via Debatescited by 19
  3. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  4. 2024Kubrick: Multimodal Agent Collaborations for Synthetic Video Generationcited by 16
  5. 2024Training Language Models to Critique With Multi-agent Feedbackcited by 16
  6. 2024Bias in the Mirror: Are LLMs opinions robust to their own adversarial attacks ?cited by 8
  7. 2025Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarkscited by 7
  8. 2025MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustnesscited by 5
  9. 2025Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluationcited by 3
  10. 2025Bias-Adjusted LLM Agents for Human-Like Decision-Making via Behavioral Economicscited by 2
  11. 2025Diffusion-Based Image Editing: An Unforeseen Adversary to Robust Invisible Watermarkscited by 2
  12. 2025The Role of Model Confidence on Bias Effects in Measured Uncertainties for Vision-Language Modelscited by 1
8 more
  1. 2025DS@GT at Touch'e: Large Language Models for Retrieval-Augmented Debatecited by 1
  2. 2025CRAwDAD: Causal Reasoning Augmentation with Dual-Agent Debatecited by 1
  3. 2026DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarioscited by 1
  4. 2025FADE: Adversarial Concept Erasure in Flow Modelscited by 0
  5. 2025Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustnesscited by 0
  6. 2025Debate with Images: Detecting Deceptive Behaviors in Multimodal Large Language Modelscited by 0
  7. 2026MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluationcited by 0
  8. 2026DS@GT ARC at Touch'e: Large Language Models for Retrieval-Augmented Debatecited by 0