mlsanityData to 7 Oct 2026

Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs

arXiv 2311.17371 ↗First posted 29 Nov 2023cs.CL cs.AIcited by 103

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Recent advancements in large language models (LLMs) underscore their potential for responding to inquiries in various domains. However, ensuring that generative agents provide accurate and reliable answers remains an ongoing challenge. In this context, multi-agent debate (MAD) has emerged as a promising strategy for enhancing the truthfulness of LLMs. We benchmark a range of debating and prompting strategies to explore the trade-offs between cost, time, and accuracy. Importantly, we find that multi-agent debating systems, in their current form, do not reliably outperform other proposed prompting strategies, such as self-consistency and ensembling using multiple reasoning paths. However, when performing hyperparameter tuning, several MAD systems, such as Multi-Persona, perform better. This suggests that MAD protocols might not be inherently worse than other approaches, but that they are more sensitive to different hyperparameter settings and difficult to optimize. We build on these results to offer insights into improving debating strategies, such as adjusting agent agreement levels, which can significantly enhance performance and even surpass all other non-debate protocols we evaluated. We provide an open-source repository to the community with several state-of-the-art protocols together with evaluation scripts to benchmark across popular research datasets.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 26 of its 35 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

26 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May
  2. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr

Other references · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  11. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
12 more
  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2022Galactica: A Large Language Model for Sciencecited by 736
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  4. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  5. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  6. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  7. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  8. 2022Can large language models reason about medical questions?cited by 194
  9. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  10. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 85
  11. 2023Learning gain differences between ChatGPT and human tutor generated algebra hintscited by 42
  12. 2023Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perceptioncited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 27; 2026: 67.

103 papers.

Sort:

By area:

  1. 2025Towards a Science of Scaling Agent Systemscited by 125
  2. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?cited by 74
  3. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  4. 2025Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneitycited by 43
  5. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  6. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  7. 2023Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate Systemcited by 26
  8. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 22
  9. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  10. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  11. 2024Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applicationscited by 19
  12. 2025Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectivenesscited by 18
38 more
  1. 2025MALLM: Multi-Agent Large Language Models Frameworkcited by 15
  2. 2025Byzantine-Robust Decentralized Coordination of LLM Agentscited by 13
  3. 2025MARS: toward more efficient multi-agent collaboration for LLM reasoningcited by 13
  4. 2025Enhancing LLM Performance Through Debate: An Empirical Study on Multi-Agent Debate for Coding Taskscited by 11
  5. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMscited by 11
  6. 2026MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systemscited by 11
  7. 2025Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluationcited by 10
  8. 2026DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factualitycited by 10
  9. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  10. 2024Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Frameworkcited by 8
  11. 2025LLM-Net: Democratizing LLMs-as-a-Service through Blockchain-based Expert Networkscited by 7
  12. 2025RedDebate: Safer Responses Through Multi-Agent Red Teaming Debatescited by 7
  13. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  14. 2024EVINCE: Optimizing Multi-LLM Dialogues Using Conditional Statistics and Information Theorycited by 6
  15. 2026Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debatecited by 6
  16. 2025From Evidence to Verdict: An Agent-Based Forensic Framework for AI-Generated Image Detectioncited by 5
  17. 2026DynaDebate: Breaking Homogeneity in Multi-Agent Debate with Dynamic Path Generationcited by 5
  18. 2026Heterogeneous Debate Engine: Identity-Grounded Cognitive Architecture for Resilient LLM-Based Ethical Tutoringcited by 5
  19. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  20. 2025OPTAGENT: Optimizing Multi-Agent LLM Interactions Through Verbal Reinforcement Learning for Enhanced Reasoningcited by 4
  21. 2025When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?cited by 4
  22. 2026More Rounds, More Noise: Why Multi-Turn Review Fails to Improve Cross-Context Verificationcited by 4
  23. 2026Hear Both Sides: Efficient Multi-Agent Debate via Diversity-Aware Message Retentioncited by 4
  24. 2026Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verificationcited by 4
  25. 2026Scaling Discovery through Test-Time Communicationcited by 4
  26. 2025The Effect of Belief Boxes and Open-mindedness on Persuasioncited by 3
  27. 2026Yes FLoReNce, I Will Do Better Next Time! Agentic Feedback Reasoning for Humorous Meme Detectioncited by 3
  28. 2026Representational Collapse in Multi-Agent LLM Committees: Measurement and Diversity-Aware Consensuscited by 3
  29. 2026EquiMem: Calibrating Shared Memory in Multi-Agent Debate via Game-Theoretic Equilibriumcited by 3
  30. 2026Beyond Consensus: Trace-Level Synthesis in Mixture of Agentscited by 3
  31. 2026The Ringelmann Effect in Multi-Agent LLM Systems: A Scaling Law for Effective Team Sizecited by 3
  32. 2026Multi-Agent Debate Strategies: Survey, Taxonomy, and Challengescited by 3
  33. 2026When Truth Is Distributed: Misinformation Derails Collective Fact Recovery in LLM-Based Multi-Agent Systemscited by 3
  34. 2025Self Iterative Label Refinement via Robust Unlabeled Learningcited by 2
  35. 2026Context Learning for Multi-Agent Discussioncited by 2
  36. 2026Small Agent Group is the Future of Digital Healthcited by 2
  37. 2026Breaking the Martingale Curse: Multi-Agent Debate via Asymmetric Cognitive Potential Energycited by 2
  38. 2026Collective Alignment in LLM Multi-Agent Systems: Disentangling Bias from Cooperation via Statistical Physicscited by 2

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversity
  2. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
  3. 2025Enhancing Multi-Agent Debate System Performance via Confidence Expression
  4. 2025Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning
  5. 2025S2-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiency
  6. 2025Self-signals Driven Multi-LLM Debate for Efficient and Accurate Reasoning
  7. 2025Free-MAD: Consensus-Free Multi-Agent Debate
  8. 2025iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inference