# Can LLMs Judge Debates? Evaluating Non-Linear Reasoning via Argumentation Theory Semantics

arXiv: https://arxiv.org/abs/2509.15739 · HTML: https://mlsanity.com/paper/2509.15739/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 6 of its 25 references that are in the map.
- First posted on arXiv 19 Sep 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 6 of 6)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Memorization in In-Context Learning](https://mlsanity.com/paper/2408.11546/) (2024)
- [Can formal argumentative reasoning enhance LLMs performances?](https://mlsanity.com/paper/2405.13036/) (2024)

## Built on by (most cited first, 5 of 5)

- [BLUEmed: Retrieval-Augmented Multi-Agent Debate for Clinical Error Detection](https://mlsanity.com/paper/2604.10389/) (2026)
- [GRASP: Deterministic argument ranking in interaction graphs](https://mlsanity.com/paper/2605.19141/) (2026)
- [A Theory of Post-hoc Debate Judgement](https://mlsanity.com/paper/2608.19002/) (2026)
- [Counting Moves, Weighing Voices: Bayesian Dialectical Argumentation for Calibrated Multi-LLM Councils under Persistent Adversaries](https://mlsanity.com/paper/2610.02005/) (2026)
- [Evidence-First Agentic Verification](https://mlsanity.com/paper/2610.17724/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) excel at linear reasoning tasks but remain underexplored on non-linear structures such as those found in natural debates, which are best expressed as argument graphs. We evaluate whether LLMs can approximate structured reasoning from Computational Argumentation Theory (CAT). Specifically, we use Quantitative Argumentation Debate (QuAD) semantics, which assigns acceptability scores to arguments based on their attack and support relations. Given only dialogue-formatted debates from two NoDE datasets, models are prompted to rank arguments without access to the underlying graph. We test several LLMs under advanced instruction strategies, including Chain-of-Thought and In-Context Learning. While models show moderate alignment with QuAD rankings, performance degrades with longer inputs or disrupted discourse flow. Advanced prompting helps mitigate these effects by reducing biases related to argument length and position. Our findings highlight both the promise and limitations of LLMs in modeling formal argumentation semantics and motivate future work on graph-aware reasoning.