mlsanityData to 7 Oct 2026

ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs

arXiv 2309.13007 ↗First posted 22 Sep 2023cs.CL cs.AI cs.LGcited by 297

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large Language Models (LLMs) still struggle with natural language reasoning tasks. Motivated by the society of minds (Minsky, 1988), we propose ReConcile, a multi-model multi-agent framework designed as a round table conference among diverse LLM agents. ReConcile enhances collaborative reasoning between LLM agents via multiple rounds of discussion, learning to convince other agents to improve their answers, and employing a confidence-weighted voting mechanism that leads to a better consensus. In each round, ReConcile initiates discussion between agents via a 'discussion prompt' that consists of (a) grouped answers and explanations generated by each agent in the previous round, (b) their confidence scores, and (c) demonstrations of answer-rectifying human explanations, used for convincing other agents. Experiments on seven benchmarks demonstrate that ReConcile significantly improves LLMs' reasoning -- both individually and as a team -- surpassing prior single-agent and multi-agent baselines by up to 11.4% and even outperforming GPT-4 on three datasets. ReConcile also flexibly incorporates different combinations of agents, including API-based, open-source, and domain-specific models, leading to an 8% improvement on MATH. Finally, we analyze the individual components of ReConcile, demonstrating that the diversity originating from different models is critical to its superior performance. Code: https://github.com/dinobby/ReConcile

Key facts. Cited by 297 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 48 of its 70 references that are in the map. First posted on arXiv 22 Sep 2023.

Builds on

48 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 47

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2017On Calibration of Modern Neural Networkscited by 4,860
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
35 more
  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  13. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  14. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  15. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  16. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  17. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  18. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  19. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  20. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  21. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  22. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  23. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  24. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  25. 2022Large Language Models Are Reasoning Teacherscited by 318
  26. 2022Teaching Small Language Models to Reasoncited by 279
  27. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  28. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  29. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  30. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 105
  31. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  32. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  33. 2023Metacognitive Prompting Improves Understanding in Large Language Modelscited by 58
  34. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  35. 2022Composing Ensembles of Pre-trained Models via Iterative Consensuscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 55; 2025: 123; 2026: 108.

297 papers.

Sort:

By area:

  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  2. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  3. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  4. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  5. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  6. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  7. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  8. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  9. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  10. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  11. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 153
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
38 more
  1. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  2. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  3. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  4. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  5. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  6. 2025Agentic Large Language Models, a surveycited by 105
  7. 2024Improving Multi-Agent Debate with Sparse Communication Topologycited by 103
  8. 2024AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planningcited by 81
  9. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  10. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  11. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?cited by 74
  12. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 63
  13. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  14. 2025Voting or Consensus? Decision-Making in Multi-Agent Debatecited by 55
  15. 2025Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?cited by 52
  16. 2023Multi-Agent Consensus Seeking via Large Language Modelscited by 50
  17. 2024StreamBench: Towards Benchmarking Continuous Improvement of Language Agentscited by 44
  18. 2025Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneitycited by 43
  19. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 41
  20. 2024MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoningcited by 40
  21. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  22. 2023The Consensus Game: Language Model Generation via Equilibrium Searchcited by 35
  23. 2024MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoningcited by 34
  24. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  25. 2024Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statementscited by 33
  26. 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMscited by 33
  27. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attentioncited by 32
  28. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  29. 2024Reverse Thinking Makes LLMs Stronger Reasonerscited by 29
  30. 2025The Truth Becomes Clearer Through Debate! Multi-Agent Systems with Large Language Models Unmask Fake Newscited by 29
  31. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 29
  32. 2025AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answeringcited by 29
  33. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  34. 2025LLMs Can Generate a Better Answer by Aggregating Their Own Responsescited by 27
  35. 2025Free-MAD: Consensus-Free Multi-Agent Debatecited by 27
  36. 2024Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberationcited by 26
  37. 2025MAS-ZERO: Designing Multi-Agent Systems with Zero Supervisioncited by 26
  38. 2024Soft Self-Consistency Improves Language Model Agentscited by 25

50 most cited of 297 All 297 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration
  2. 2025Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning
  3. 2025MARS: toward more efficient multi-agent collaboration for LLM reasoning
  4. 2025Adaptive Coopetition: Leveraging Coarse Verifier Signals for Resilient Multi-Agent LLM Reasoning
  5. 2026A Single Revision Step Improves Token-Efficient LLM Reasoning
  6. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
  7. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
  8. 2025Self-Improvement of Language Models by Post-Training on Multi-Agent Debate