mlsanityData to 14 Oct 2026

What Does Multi-Agent LLM Debate Actually Change? A Layered Analysis of Disagreement and Answer Quality

arXiv 2609.08016 ↗First posted 7 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Multi-agent debate, in which several LLMs exchange arguments before producing an answer, raises a basic question: does expressed disagreement reflect changes in the members' own positions? No single signal can settle this question, so we organize the analysis around five questions: (A) does the debater say it disagrees; (B) does its reply text actually argue; (C) does its own position change after each debate turn; (D) how much, quantitatively, does the position change; and (E) how do members' final positions compare with their initial ones? We evaluate two- and three-member committees on 50 curated opinion questions from GlobalOpinionQA, using same-model, same-family, and mixed-family configurations with friendly, neutral, and hostile instructions assigned at each turn. (A) Tone changes reported agreement: the share of replies reporting strong agreement is 70.7-96.3% under friendly instructions, compared with 9.5-19.3% under hostile ones, varying with model choice. (B) Self-reports broadly align with text judgments, but consistency varies by agreement level and model choice. (C) Position changes depend on the interaction: after a peer's leaning-disagree reply, members reporting strong agreement switch options more often than those reporting leaning disagreement. (D) For open-weight members, the probability of the option a member already holds stays near saturation, even after a peer's pushback, while endorsement of its earlier position text drops after a peer's argument relative to neutral filler, more for Qwen3.8-27B than Inkling. (E) The selected option is unchanged from members' initial to final positions in over 90% of comparisons in every configuration. Taken together, expressed disagreement need not translate into position revision, either within individual exchanges or over a complete debate.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 46 of its 90 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

46 of 90 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge9 Jun
  2. 2026Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate30 May
  3. 2026Too Polite to Disagree: Understanding Sycophancy Propagation in Multi-Agent Systems3 Apr
  4. 2025Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate27 Sep
  5. 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate5 Sep
  6. 2025Stay Focused: Problem Drift in Multi-Agent Debate26 Feb
  7. 2025Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity12 Feb
  8. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs29 Nov
  9. 2023Simulating Opinion Dynamics with Networks of LLM-based Agents16 Nov
  10. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 36

  1. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  8. 2023Whose Opinions Do Language Models Reflect?cited by 773
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  10. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  11. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  12. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
24 more
  1. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  2. 2018AI safety via debatecited by 400
  3. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  4. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  5. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  6. 2024More Agents Is All You Needcited by 180
  7. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 159
  8. 2024Emergent social conventions and collective bias in LLM populationscited by 123
  9. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  10. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 109
  11. 2025SycEval: Evaluating LLM Sycophancycited by 108
  12. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  13. 2023Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debatecited by 97
  14. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  15. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 54
  16. 2023When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Modelscited by 53
  17. 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 53
  18. 2023Debate Helps Supervise Unreliable Expertscited by 43
  19. 2026Multi-Agent Teams Hold Experts Backcited by 21
  20. 2024Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Taskscited by 18
  21. 2024DEBATE: Devil's Advocate-Based Assessment and Text Evaluationcited by 16
  22. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 13
  23. 2026It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertaintycited by 6
  24. 2026Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scalingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
  2. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
  3. 2026The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate
  4. 2026The impact of multi-agent debate protocols on debate quality: a controlled case study
  5. 2025When Persuasion Overrides Truth in Multi-Agent LLM Debates: Introducing a Confidence-Weighted Persuasion Override Rate (CW-POR)
  6. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
  7. 2026Multi-agent discussion gains less when dissent is withheld
  8. 2025When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?