mlsanityData to 14 Oct 2026

Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

arXiv 2305.19118 ↗First posted 30 May 2023cs.CLcited by 1,011

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Modern large language models (LLMs) like ChatGPT have shown remarkable performance on general language tasks but still struggle on complex reasoning tasks, which drives the research on cognitive behaviors of LLMs to explore human-like problem-solving strategies. Along this direction, one representative strategy is self-reflection, which asks an LLM to refine the solution with the feedback generated by itself iteratively. However, our study shows that such reflection-style methods suffer from the Degeneration-of-Thought (DoT) problem: once the LLM has established confidence in its solutions, it is unable to generate novel thoughts later through reflection even if its initial stance is incorrect. To address the DoT problem, we propose a Multi-Agent Debate (MAD) framework, in which multiple agents express their arguments in the state of "tit for tat" and a judge manages the debate process to obtain a final solution. Clearly, our MAD framework encourages divergent thinking in LLMs which would be helpful for tasks that require deep levels of contemplation. Experiment results on two challenging datasets, commonsense machine translation and counter-intuitive arithmetic reasoning, demonstrate the effectiveness of our MAD framework. Extensive analyses suggest that the adaptive break of debate and the modest level of "tit for tat" state are required for MAD to obtain good performance. Moreover, we find that LLMs might not be a fair judge if different LLMs are used for agents. Code is available at https://github.com/Skytliang/Multi-Agents-Debate.

Key facts. Cited by 1,011 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 31 of its 41 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

31 of 41 references.

References · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
19 more
  1. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  4. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  5. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  6. 2016Solving General Arithmetic Word Problemscited by 360
  7. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  8. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 289
  9. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 207
  10. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  11. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  12. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  13. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  14. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 77
  15. 2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
  16. 2023Interactive-Chain-Prompting: Ambiguity Resolution for Crosslingual Conditional Generation with Interactioncited by 16
  17. 2023Question Answering as Programming for Solving Time-Sensitive Questionscited by 14
  18. 2020Meta-CoTGAN: A Meta Cooperative Training Paradigm for Improving Adversarial Text Generationcited by 11
  19. 2021Eliciting Thinking Hierarchy without a Priorcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 60; 2024: 186; 2025: 359; 2026: 406.

1,011 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  5. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  7. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  9. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  10. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  11. 2023Cognitive Architectures for Language Agentscited by 429
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
38 more
  1. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  2. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  3. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  4. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  5. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  6. 2023Large Language Model Alignment: A Surveycited by 243
  7. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 241
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  9. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  10. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  11. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 222
  12. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 216
  13. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  14. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  15. 2024More Agents Is All You Needcited by 180
  16. 2025Multi-agent Architecture Search via Agentic Supernetcited by 179
  17. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  18. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
  19. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 159
  20. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  21. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  22. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 145
  23. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 145
  24. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 141
  25. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 133
  26. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 132
  27. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  28. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  29. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  30. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 122
  31. 2024LLM Multi-Agent Systems: Challenges and Open Problemscited by 120
  32. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 120
  33. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 119
  34. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  35. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 117
  36. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  37. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  38. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 114

50 most cited of 1,011 All 1,011 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diversity of Thought Elicits Stronger Reasoning Capabilities in Multi-Agent Debate Frameworks
  2. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
  3. 2024Mirror: A Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoning
  4. 2026Multi-Agent Debate with Memory Masking
  5. 2025THiNK: Can Large Language Models Think-aloud?
  6. 2026Reasoning Models Generate Societies of Thought
  7. 2025MetaCrit: A Critical Thinking Framework for Self-Regulated LLM Reasoning
  8. 2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debate