mlsanityData to 14 Oct 2026

Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate

arXiv 2305.11595 ↗First posted 19 May 2023cs.CL cs.AIcited by 97

Not on the map. Why?

Cited by 59 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have shown impressive capabilities in various applications, but they still face various inconsistency issues. Existing works primarily focus on the inconsistency issues within a single LLM, while we complementarily explore the inter-consistency among multiple LLMs for collaboration. To examine whether LLMs can collaborate effectively to achieve a consensus for a shared goal, we focus on commonsense reasoning, and introduce a formal debate framework (FORD) to conduct a three-stage debate among LLMs with real-world scenarios alignment: fair debate, mismatched debate, and roundtable debate. Through extensive experiments on various datasets, LLMs can effectively collaborate to reach a consensus despite noticeable inter-inconsistencies, but imbalances in their abilities can lead to domination by superior LLMs. Leveraging a more advanced LLM like GPT-4 as an authoritative judge can boost collaboration performance. Our work contributes to understanding the inter-consistency among LLMs and lays the foundation for developing future collaboration methods. Codes and data are available at https://github.com/Waste-Wood/FORD

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

22 of 40 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
10 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  4. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  6. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  7. 2019Abductive Commonsense Reasoningcited by 301
  8. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  9. 2021Explaining Answers with Entailment Treescited by 152
  10. 2022Unpacking Large Language Models with Conceptual Consistencycited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 24; 2025: 45; 2026: 24.

97 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  4. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  5. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 206
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  7. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attackscited by 122
  8. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 82
  9. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?cited by 81
  10. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  11. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  12. 2024GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussioncited by 67
38 more
  1. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  2. 2025Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneitycited by 44
  3. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  4. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  5. 2025Free-MAD: Consensus-Free Multi-Agent Debatecited by 29
  6. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  7. 2024Multi-Agent Causal Discovery Using Large Language Modelscited by 26
  8. 2025Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skillscited by 24
  9. 2025S2-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiencycited by 22
  10. 2024Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Frameworkcited by 19
  11. 2024Evaluating the Performance of Large Language Models via Debatescited by 19
  12. 2024Multi-Agents are Social Groups: Investigating Social Influence of Multiple Agents in Human-Agent Interactionscited by 16
  13. 2025Literature Review Of Multi-Agent Debate For Problem-Solvingcited by 15
  14. 2024ERD: A Framework for Improving LLM Reasoning for Cognitive Distortion Classificationcited by 14
  15. 2024An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Makingcited by 14
  16. 2025Byzantine-Robust Decentralized Coordination of LLM Agentscited by 14
  17. 2025Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMscited by 14
  18. 2025MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systemscited by 13
  19. 2024Self-Prompt Tuning: Enable Autonomous Role-Playing in LLMscited by 12
  20. 2025AgentGroupChat-V2: Divide-and-Conquer Is What LLM-Based Multi-Agent System Needcited by 12
  21. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMscited by 12
  22. 2024Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discoverycited by 11
  23. 2025Enhancing LLM Performance Through Debate: An Empirical Study on Multi-Agent Debate for Coding Taskscited by 11
  24. 2025Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancementcited by 10
  25. 2025MAD-Spear: A Conformity-Driven Prompt Injection Attack on Multi-Agent Debate Systemscited by 10
  26. 2025DRF: LLM-AGENT Dynamic Reputation Filtering Frameworkcited by 10
  27. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
  28. 2024Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaborationcited by 9
  29. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 9
  30. 2026MetaGen: Self-Evolving Roles and Topologies for Multi-Agent LLM Reasoningcited by 9
  31. 2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learningcited by 9
  32. 2024Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidancecited by 8
  33. 2025Multi-Agent Debate Strategies to Enhance Requirements Engineering with Large Language Modelscited by 8
  34. 2025CortexDebate: Debating Sparsely and Equally for Multi-Agent Debatecited by 7
  35. 2023Playing Large Games with Oracles and AI Debatecited by 6
  36. 2024ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Modelcited by 5
  37. 2025Agent-to-Agent Theory of Mind: Testing Interlocutor Awareness among Large Language Modelscited by 5
  38. 2023SAIE Framework: Support Alone Isn't Enough -- Advancing LLM Training with Adversarial Remarkscited by 4

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Single Models: Enhancing LLM Detection of Ambiguity in Requests through Debate
  2. 2024Evaluating Consistency and Reasoning Capabilities of Large Language Models
  3. 2025Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth
  4. 2025Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning
  5. 2025Ensemble Debates with Local Large Language Models for AI Alignment
  6. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
  7. 2025Modeling Response Consistency in Multi-Agent LLM Systems: A Comparative Analysis of Shared and Separate Context Approaches
  8. 2025Cross-linguistic disagreement as a conflict of semantic alignment norms in multilingual AI~Linguistic Diversity as a Problem for Philosophy, Cognitive Science, and AI~