mlsanityData to 7 Oct 2026

Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration

arXiv 2311.08152 ↗First posted 14 Nov 2023cs.CLcited by 48

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable capabilities in general natural language processing tasks but often fall short in complex reasoning tasks. Recent studies have explored human-like problem-solving strategies, such as self-correct, to push further the boundary of single-model reasoning ability. In this work, we let a single model "step outside the box" by engaging multiple models to correct each other. We introduce a multi-agent collaboration strategy that emulates the academic peer review process. Each agent independently constructs its own solution, provides reviews on the solutions of others, and assigns confidence levels to its reviews. Upon receiving peer reviews, agents revise their initial solutions. Extensive experiments on three different types of reasoning tasks show that our collaboration approach delivers superior accuracy across all ten datasets compared to existing methods. Further study underscores the effectiveness of integrating confidence in reviews, demonstrates the superiority of feedback exchange over mere solution sharing, and highlights the role of capability and diversity in fostering successful collaboration.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 23 of its 38 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

23 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration30 Sep
  2. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
9 more
  1. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  2. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  4. 2023Language Models can Solve Computer Taskscited by 371
  5. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  6. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  7. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  8. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  9. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 29; 2026: 9.

48 papers.

Sort:

By area:

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  5. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  7. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  8. 2024GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussioncited by 64
  9. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  10. 2024Mora: Enabling Generalist Video Generation via A Multi-Agent Frameworkcited by 41
  11. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  12. 2023Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate Systemcited by 26
36 more
  1. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  2. 2024Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Taskscited by 22
  3. 2025Group Think: Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularitycited by 22
  4. 2025S2-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiencycited by 21
  5. 2026Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judgecited by 16
  6. 2025A Unified Agentic Framework for Evaluating Conditional Image Generationcited by 15
  7. 2025ComfyUI-R1: Exploring Reasoning Models for Workflow Generationcited by 15
  8. 2024An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Makingcited by 14
  9. 2025Augmenting Multi-Agent Communication with State Delta Trajectorycited by 13
  10. 2025ComfyUI-Copilot: An Intelligent Assistant for Automated Workflow Developmentcited by 11
  11. 2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judgecited by 10
  12. 2025Multi-LLM Collaborative Search for Complex Problem Solvingcited by 9
  13. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  14. 2025Multi-Agent Debate Strategies to Enhance Requirements Engineering with Large Language Modelscited by 8
  15. 2025Position: Towards a Responsible LLM-empowered Multi-Agent Systemscited by 7
  16. 2025RMoA: Optimizing Mixture-of-Agents through Diversity Maximization and Residual Compensationcited by 7
  17. 2025CortexDebate: Debating Sparsely and Equally for Multi-Agent Debatecited by 6
  18. 2025GenoMAS: A Multi-Agent Framework for Scientific Discovery via Code-Driven Gene Expression Analysiscited by 6
  19. 2024GenoTEX: An LLM Agent Benchmark for Automated Gene Expression Data Analysiscited by 5
  20. 2025Project Riley: Multimodal Multi-Agent LLM Collaboration with Emotional Reasoning and Votingcited by 4
  21. 2026EvoMAS: Evolutionary Generation of Multi-Agent Systemscited by 4
  22. 2025The Effect of Belief Boxes and Open-mindedness on Persuasioncited by 3
  23. 2026Multi-Agent Debate Strategies: Survey, Taxonomy, and Challengescited by 3
  24. 2025AgentCDM: Enhancing Multi-Agent Collaborative Decision-Making via ACH-Inspired Structured Reasoningcited by 2
  25. 2025Can Lessons From Human Teams Be Applied to Multi-Agent Systems? The Role of Structure, Diversity, and Interaction Dynamicscited by 2
  26. 2025Interaction Protocol Shapes Moral Judgment in Multi-Agent Debatecited by 2
  27. 2026SymbOmni: Evolving Agentic Omni Models via Symbolic Concept Learningcited by 2
  28. 2025Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarizationcited by 1
  29. 2025COSMIR: Chain Orchestrated Structured Memory for Iterative Reasoning over Long Contextcited by 1
  30. 2026Learning to Seek Help: Dynamic Collaboration Between Small and Large Language Modelscited by 1
  31. 2026CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologiescited by 1
  32. 2024ReFeR: Improving Evaluation and Reasoning through Hierarchy of Modelscited by 0
  33. 2025Learning to Align Multi-Faceted Evaluation: A Unified and Robust Frameworkcited by 0
  34. 2026MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluationcited by 0
  35. 2026ArcticSwarm: Deferring Early Consensus in Long-Horizon Multi-Agent Researchcited by 0
  36. 2026Topological Coherence for Self-evolving Multi-agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MARS: toward more efficient multi-agent collaboration for LLM reasoning
  2. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
  3. 2025MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration
  4. 2025Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning
  5. 2026Can Small Agents Collaborate to Beat a Single Large Language Model?
  6. 2024Diversity of Thought Elicits Stronger Reasoning Capabilities in Multi-Agent Debate Frameworks
  7. 2025Unleashing Diverse Thinking Modes in LLMs through Multi-Agent Collaboration
  8. 2025Adaptive Reasoning Executor: A Collaborative Agent System for Efficient Reasoning