mlsanityData to 7 Oct 2026

Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systems

arXiv 2609.17306 ↗First posted 15 Sep 2026cs.MA cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Multi-agent Systems (MAS) combine multiple model outputs to solve complex reasoning tasks. However, despite rapid growth of available open-source models, there is limited research on how to select optimal model candidates out of this massive pool. We systematically evaluate 8 model selection strategies (including model size, accuracy and answer diversity) across before-generation (routing) and after-generation (majority-voting, LLM-as-a-judge) MAS architectures on challenging scientific benchmarks. Our findings show a significant gap between theoretical oracle potential and actual performance: Expanding candidate pool sizes often degrades performance below that of the top performing base-model. We find that candidate selection within a single model family is the strategy that yields the best relative performance over a standalone model. These results demonstrate that adding arbitrary models to a heterogeneous MAS can introduce system instability, highlighting model selection as a critical design choice for multi-agent systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 30 of its 64 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

30 of 64 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity3 Feb
  2. 2025Towards a Science of Scaling Agent Systems9 Dec
  3. 2024Mixture-of-Agents Enhances Large Language Model Capabilities7 Jun
  4. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration3 Oct
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar

Other references · 25

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  6. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  7. 2025Humanity's Last Examcited by 467
  8. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  9. 2025BIG-Bench Extra Hardcited by 123
  10. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 103
  11. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  12. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 49
13 more
  1. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  2. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 43
  3. 2026FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Taskscited by 42
  4. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  5. 2025Advancing Expert Specialization for Better MoEcited by 37
  6. 2026LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routingcited by 35
  7. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  8. 2025Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluationcited by 29
  9. 2025ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledgecited by 20
  10. 2025GenSelect: A Generative Approach to Best-of-Ncited by 14
  11. 2025Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Routercited by 13
  12. 2026Reasoning Models Generate Societies of Thoughtcited by 11
  13. 2024Bench-CoE: a Framework for Collaboration of Experts from Benchmarkcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Orchestrating Intelligence: Confidence-Aware Routing for Efficient Multi-Agent Collaboration across Multi-Scale Models
  2. 2026MASEval: Extending Multi-Agent Evaluation from Models to Systems
  3. 2025MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision
  4. 2026When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines
  5. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies
  6. 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMs
  7. 2025Optimizing Model Selection for Compound AI Systems
  8. 2025MasRouter: Learning to Route LLMs for Multi-Agent Systems