# Multi-Agent Debate for LLM Judges with Adaptive Stability Detection

arXiv: https://arxiv.org/abs/2510.12697 · HTML: https://mlsanity.com/paper/2510.12697/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 50 of its 71 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 30 of 50)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://mlsanity.com/paper/2401.10019/) (2024)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Benchmarking Cognitive Biases in Large Language Models as Evaluators](https://mlsanity.com/paper/2309.17012/) (2023)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)

## Built on by (most cited first, 23 of 23)

- [From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation](https://mlsanity.com/paper/2604.07667/) (2026)
- [APE: Agentic Prompt Enhancer for Image Generation and Editing](https://mlsanity.com/paper/2606.00204/) (2026)
- [SLMJury: Can Small Language Models Judge as Well as Large Ones?](https://mlsanity.com/paper/2606.07810/) (2026)
- [XBridge: Entity-Grounded Latent Bridge for Heterogeneous LLM Communication](https://mlsanity.com/paper/2608.11676/) (2026)
- [Belief Engine: Configurable and Inspectable Stance Dynamics in Multi-Agent LLM Deliberation](https://mlsanity.com/paper/2605.15343/) (2026)
- [Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts](https://mlsanity.com/paper/2512.03704/) (2025)
- [CSE-UOI at SemEval-2026 Task 6: A Two-Stage Heterogeneous Ensemble with Deliberative Complexity Gating for Political Evasion Detection](https://mlsanity.com/paper/2603.12453/) (2026)
- [PaperJury: Due-Process Review for Bounded LaTeX Revision](https://mlsanity.com/paper/2606.16322/) (2026)
- [Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds](https://mlsanity.com/paper/2606.29654/) (2026)
- [BLUEmed: Retrieval-Augmented Multi-Agent Debate for Clinical Error Detection](https://mlsanity.com/paper/2604.10389/) (2026)
- [MDIA: A Multi-Agent Diagnostic Intelligence Pipeline on HealthBench Professional](https://mlsanity.com/paper/2605.24699/) (2026)
- [Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages](https://mlsanity.com/paper/2605.30529/) (2026)
- [SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations](https://mlsanity.com/paper/2606.05563/) (2026)
- [MoCA-Agent: A Market-of-Claims Code Agent for Financial and Numerical Reasoning](https://mlsanity.com/paper/2606.11537/) (2026)
- [Divergent-Convergent Reasoning: Scaling Test-Time Compute through Structured Solution Synthesis](https://mlsanity.com/paper/2608.15303/) (2026)
- [A Theory of Post-hoc Debate Judgement](https://mlsanity.com/paper/2608.19002/) (2026)
- [Meta-Moderator: Empowering Multi-Agent Debate with Meta-Cognition](https://mlsanity.com/paper/2608.23029/) (2026)
- [MABPD: Multi-Agent Bias Probing & Detection via Structured Argument Debate](https://mlsanity.com/paper/2609.04841/) (2026)
- [CERA-MoA: Co-Evolving Routing Mechanisms with Continually Learning LLM Agents](https://mlsanity.com/paper/2609.18779/) (2026)
- [MiniRep: Robust Reputation-Based Aggregation for Multi-Agent Debate](https://mlsanity.com/paper/2609.39297/) (2026)
- [DUET: Co-Evolving Solver and Grader Agents](https://mlsanity.com/paper/2610.04087/) (2026)
- [Efficient Multi-Agent Reasoning via Confidence-Guided Adaptive Debate](https://mlsanity.com/paper/2610.16343/) (2026)
- [PFedMA: Who Should Join Next as Federated Multi-Agent Orchestration Evolves?](https://mlsanity.com/paper/2610.19458/) (2026)

## Abstract (arXiv)

With advancements in reasoning capabilities, Large Language Models (LLMs) are increasingly employed for automated judgment tasks. While LLMs-as-Judges offer promise in automating evaluations, current approaches often rely on simplistic aggregation methods (e.g., majority voting), which can fail even when individual agents provide correct answers. To address this, we propose a multi-agent debate judge framework where agents collaboratively reason and iteratively refine their responses. We formalize the debate process mathematically, analyzing agent interactions and proving that debate amplifies correctness compared to static ensembles. To enhance efficiency, we introduce a stability detection mechanism that models judge consensus dynamics via a time-varying Beta-Binomial mixture, with adaptive stopping based on distributional similarity (Kolmogorov-Smirnov test). This mechanism models the judges' collective correct rate dynamics using a time-varying mixture of Beta-Binomial distributions and employs an adaptive stopping criterion based on distributional similarity (Kolmogorov-Smirnov statistic). Experiments across multiple benchmarks and models demonstrate that our framework improves judgment accuracy over majority voting while maintaining computational efficiency.