mlsanityData to 14 Oct 2026

Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity

arXiv 2502.08788 ↗First posted 12 Feb 2025cs.CL cs.LGcited by 44

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Multi-agent debate (MAD) has gained significant attention as a promising line of research to improve the factual accuracy and reasoning capabilities of large language models (LLMs). Despite its conceptual appeal, current MAD research suffers from critical limitations in evaluation practices, including limited benchmark coverage, weak baseline comparisons, and inconsistent setups. This paper presents a systematic evaluation of 5 representative MAD methods across 9 benchmarks using 4 foundational models. Surprisingly, our findings reveal that MAD often fail to outperform simple single-agent baselines such as Chain-of-Thought and Self-Consistency, even when consuming significantly more inference-time computation. To advance MAD research, we further explore the role of model heterogeneity and find it as a universal antidote to consistently improve current MAD frameworks. Based on our findings, we argue that the field must stop overvaluing MAD in its current form; for true advancement, we must critically rethink evaluation paradigms and actively embrace model heterogeneity as a core design principle.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 41 of its 46 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

41 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Improving Multi-Agent Debate with Sparse Communication Topology17 Jun
  2. 2024CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solving26 Apr
  3. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?28 Feb
  4. 2023Let Models Speak Ciphers: Multiagent Debate through Embeddings10 Oct
  5. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs22 Sep
  6. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
23 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  6. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  9. 2020Aligning AI With Shared Human Valuescited by 723
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  11. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  12. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  13. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  14. 2016Solving General Arithmetic Word Problemscited by 360
  15. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  16. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 241
  17. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 222
  18. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 114
  19. 2023Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debatecited by 97
  20. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 89
  21. 2024Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligencecited by 86
  22. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 82
  23. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 31.

44 papers.

Sort:

By area:

  1. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 38
  2. 2025MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Taskscited by 29
  3. 2025Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debatecited by 29
  4. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoningcited by 29
  5. 2025When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoningcited by 21
  6. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  7. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 10
  8. 2025iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inferencecited by 10
  9. 2026The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systemscited by 9
  10. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  11. 2026Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debatecited by 7
  12. 2026Scaling Discovery through Test-Time Communicationcited by 7
32 more
  1. 2026Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluationcited by 6
  2. 2026Epistemic Gain, Aleatoric Cost: Uncertainty Decomposition in Multi-Agent Debate for Math Reasoningcited by 5
  3. 2025Interaction Protocol Shapes Moral Judgment in Multi-Agent Debatecited by 3
  4. 2026WebWeaver: Breaking Topology Confidentiality in LLM Multi-Agent Systems with Stealthy Context-Based Inferencecited by 3
  5. 2026The Alpha Illusion: Reported Alpha from LLM Trading Agents Should Not Be Treated as Deployment Evidencecited by 3
  6. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?cited by 3
  7. 2026The Consistency Illusion: How Multi-Agent Debate Hides Reasoning Misalignmentcited by 3
  8. 2025Swarm Intelligence Enhanced Reasoning: A Density-Driven Framework for LLM-Based Multi-Agent Optimizationcited by 2
  9. 2026HCP-MAD:Heterogeneous Consensus-Progressive Reasoning for Efficient Multi-Agent Debatecited by 2
  10. 2026The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM Reasoningcited by 2
  11. 2025PanelTR: Zero-Shot Table Reasoning Framework Through Multi-Agent Scientific Discussioncited by 1
  12. 2026Multimodal Collaborative Debate for Zero-Shot Time Series Reasoningcited by 1
  13. 2026Dynamic Coordination Strategy Selection for Enterprise Multi-Agent Systemscited by 1
  14. 2026Does Multi-Agent Debate Improve AI Feedback on Research Papers?cited by 1
  15. 2026AgentPanel: Toward a New Paradigm for Human--AI Collaboration in Exploring Scientific Questionscited by 1
  16. 2026Self-Organizing Agent Teams Learn to Reason Togethercited by 1
  17. 2025Adaptive Reasoning Executor: A Collaborative Agent System for Efficient Reasoningcited by 0
  18. 2026Towards a Science of Collective AI: LLM-based Multi-Agent Systems Need a Transition from Blind Trial-and-Error to Rigorous Sciencecited by 0
  19. 2026Multi-agent decision making: A Blackwell's informativeness approachcited by 0
  20. 2026The Agent Use of Agent Beings: Agent Cybernetics Is the Missing Science of Foundation Agentscited by 0
  21. 2026CHAL: Council of Hierarchical Agentic Languagecited by 0
  22. 2026Multi-Agent Systems are Mixtures of Experts: Who Becomes an Influencer?cited by 0
  23. 2026When Helping Hurts and How to Fix It: Multi-Agent Debate for Data Cleaningcited by 0
  24. 2026TRACE: An Operational Reasoning Schema for Auditable Agentic Commitmentscited by 0
  25. 2026Process-Constituted Intelligence: A Shared Criterion for Humans and Machinescited by 0
  26. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0
  27. 2026What Does Multi-Agent LLM Debate Actually Change? A Layered Analysis of Disagreement and Answer Qualitycited by 0
  28. 2026How does Adversarial Influence Scale in Multi-Agent Systems?cited by 0
  29. 2026BaRe-Mem: Bayesian Reliability Memory for Robust and Adaptive Agent Consultationcited by 0
  30. 2026Multi-agent discussion gains less when dissent is withheldcited by 0
  31. 2026Towards Better Exploration in Sequential Test-Time Scalingcited by 0
  32. 2026When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges
  2. 2025Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness
  3. 2026M3MAD-Bench: Multi-Dimensional Evaluation of Multi-Agent Debate Across Domains and Modalities
  4. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
  5. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
  6. 2025When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?
  7. 2025Free-MAD: Consensus-Free Multi-Agent Debate
  8. 2026ARMOR-MAD: Adaptive Routing for Heterogeneous Multi-Agent Debate in Large Language Model Reasoning