mlsanityData to 14 Oct 2026

Improving Factuality and Reasoning in Language Models through Multiagent Debate

arXiv 2305.14325 ↗First posted 23 May 2023cs.CL cs.AI cs.CVcited by 1,784

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities in language generation, understanding, and few-shot learning in recent years. An extensive body of work has explored how their performance may be further improved through the tools of prompting, ranging from verification, self-consistency, or intermediate scratchpads. In this paper, we present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. We also demonstrate that our approach improves the factual validity of generated content, reducing fallacious answers and hallucinations that contemporary models are prone to. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate. Overall, our findings suggest that such "society of minds" approach has the potential to significantly advance the capabilities of LLMs and pave the way for further breakthroughs in language generation and understanding.

Key facts. Cited by 1,784 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 29 of its 48 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

29 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
16 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  8. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  9. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  10. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  11. 2018AI safety via debatecited by 400
  12. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  13. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 220
  14. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  15. 2022Composing Ensembles of Pre-trained Models via Iterative Consensuscited by 28
  16. 2022Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 107; 2024: 289; 2025: 534; 2026: 854.

1,784 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  7. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  8. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  11. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  12. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
38 more
  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  2. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  4. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  5. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  6. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  7. 2023Cognitive Architectures for Language Agentscited by 429
  8. 2024Automated Design of Agentic Systemscited by 407
  9. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  10. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  11. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  12. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  13. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  14. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  15. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  16. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  17. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  18. 2023Challenges and Applications of Large Language Modelscited by 254
  19. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  20. 2023Large Language Model Alignment: A Surveycited by 243
  21. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 241
  22. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  23. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  24. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  25. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  26. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  27. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 222
  28. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 216
  29. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 210
  30. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  31. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  32. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 206
  33. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  34. 2024TradingAgents: Multi-Agents LLM Financial Trading Frameworkcited by 188
  35. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  36. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  37. 2024More Agents Is All You Needcited by 180
  38. 2025Multi-agent Architecture Search via Agentic Supernetcited by 179

50 most cited of 1,784 All 1,784 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  2. 2026Enhancing reasoning accuracy in large language models during inference time
  3. 2025Reasoning Capabilities and Invariability of Large Language Models
  4. 2024General Purpose Verification for Chain of Thought Prompting
  5. 2024LLM2: Let Large Language Models Harness System 2 Reasoning
  6. 2024Calibrating Reasoning in Language Models with Internal Consistency
  7. 2023MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models
  8. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models