# Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarks

arXiv: https://arxiv.org/abs/2509.23537 · HTML: https://mlsanity.com/paper/2509.23537/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 7 of its 10 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 7 of 7)

- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View](https://mlsanity.com/paper/2310.02124/) (2023)
- [MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning](https://mlsanity.com/paper/2502.18439/) (2025)
- [Voting or Consensus? Decision-Making in Multi-Agent Debate](https://mlsanity.com/paper/2502.19130/) (2025)
- [From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium](https://mlsanity.com/paper/2506.08292/) (2025)
- [A Strategic Coordination Framework of Small LLMs Matches Large LLMs in Data Synthesis](https://mlsanity.com/paper/2504.12322/) (2025)

## Built on by (most cited first, 9 of 9)

- [Internet of Agentic AI: Incentive-Compatible Distributed Teaming and Workflow](https://mlsanity.com/paper/2602.03145/) (2026)
- [Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model](https://mlsanity.com/paper/2608.13867/) (2026)
- [Heterogeneous Scientific Foundation Model Collaboration](https://mlsanity.com/paper/2604.27351/) (2026)
- [OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality](https://mlsanity.com/paper/2608.05263/) (2026)
- [The ORCA Benchmark: Evaluating Real-World Calculation Accuracy in Large Language Models](https://mlsanity.com/paper/2511.02589/) (2025)
- [Agents in the Wild: Where Research Meets Deployment](https://mlsanity.com/paper/2607.19336/) (2026)
- [A MARL Centered Reference Architecture for Large Language Model Augmentation in Smart Manufacturing](https://mlsanity.com/paper/2608.07148/) (2026)
- [Mint-Agent: Introducing Finance-Native Agentic Foundation Models](https://mlsanity.com/paper/2608.16386/) (2026)
- [Right-Sizing LLM-Agent Decomposition in VAT Determination: A Pilot Controlled Sweep](https://mlsanity.com/paper/2608.23395/) (2026)

## Abstract (arXiv)

We study multi-turn multi-agent orchestration, where multiple large language model (LLM) agents interact over multiple turns by iteratively proposing answers or casting votes until reaching consensus. Using four LLMs (Gemini 2.5 Pro, GPT-5, Grok 4, and Claude Sonnet 4) on GPQA-Diamond, IFEval, and MuSR, we conduct two experiments: (i) benchmarking orchestration against single-LLM baselines; and (ii) ablations on GPQA-Diamond that vary whether agents see who authored answers and whether they can observe ongoing votes. Orchestration matches or exceeds the strongest single model and consistently outperforms the others. Analysis of best-achievable orchestration performance shows potential for further gains. The ablations show that revealing authorship increases self-voting and ties, and that showing ongoing votes amplifies herding, which speeds convergence but can sometimes yield premature consensus.