mlsanityData to 7 Oct 2026

Towards a Science of Scaling Agent Systems

arXiv 2512.08296 ↗First posted 9 Dec 2025cs.AIcited by 125

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Agents, language model-based systems capable of reasoning, planning, and acting are widely adopted in real-world tasks, yet how their performance changes as these systems scale across key dimensions remains underexplored. We introduce quantitative scaling principles for agent systems as a predictive model, capturing how performance varies with coordination, model capability, and measurable system and task factors. Across 260 configurations spanning six agentic benchmarks, five canonical architectures (Single-Agent and four Multi-Agent: Independent, Centralized, Decentralized, Hybrid), and three LLM families, we perform controlled evaluations, standardizing tools, prompts, and compute to isolate architectural effects. The resulting model achieves a cross-validated R^2=0.373 across all six benchmarks (R^2=0.413 with a task-grounded capability metric). We identify a robust capability-saturation effect and additional patterns: (1) a coordination yields diminishing returns once single-agent baselines exceed certain performance; (2) tool-heavy tasks appear to incur multi-agent overhead; and (3) architectures without centralized verification tend to propagate errors more than those with centralized coordination. Relative performance change compared to single-agent baseline ranges from +80.8% on decomposable financial reasoning to -70.0% on sequential planning, demonstrating that architecture-task alignment determines collaborative success. The framework identifies the best-performing architecture for 87% of held-out configurations and shows consistent relative architecture preferences on unseen frontier models. Agent effectiveness depends on alignment between coordination and task structure, and that mismatched coordination degrades the performance.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 53 of its 65 references that are in the map. First posted on arXiv 9 Dec 2025.

Builds on

53 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games20 Nov
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar

Other references · 51

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
38 more
  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  11. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  12. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  13. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  14. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  15. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  16. 2023Cognitive Architectures for Language Agentscited by 422
  17. 2025Accelerating scientific discovery with Co-Scientistcited by 353
  18. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  19. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  20. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  21. 2024AI Agents That Mattercited by 199
  22. 2025Survey on Evaluation of LLM-based Agentscited by 190
  23. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  24. 2025Generalizing Verifiable Instruction Followingcited by 178
  25. 2024More Agents Is All You Needcited by 174
  26. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  27. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  28. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 163
  29. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  30. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  31. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 103
  32. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 102
  33. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  34. 2025Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Taskscited by 63
  35. 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Settingcited by 47
  36. 2025Single-agent or Multi-agent Systems? Why Not Both?cited by 28
  37. 2024Plancraft: an evaluation dataset for planning with LLM agentscited by 16
  38. 2025The Anatomy of a Personal Health Agentcited by 15

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 124.

125 papers.

Sort:

By area:

  1. 2026When Single-Agent with Skills Replace Multi-Agent Systems and When They Failcited by 34
  2. 2026Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgetscited by 28
  3. 2026Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversitycited by 25
  4. 2026Rethinking the Value of Multi-Agent Workflow: A Strong Single Agent Baselinecited by 18
  5. 2026Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agentscited by 13
  6. 2026AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agentcited by 12
  7. 2026Agents in the Wild: Safety, Society, and the Illusion of Sociality on Moltbookcited by 10
  8. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  9. 2026Dual Latent Memory for Visual Multi-agent Systemcited by 9
  10. 2025An Information Theoretic Perspective on Agentic System Designcited by 8
  11. 2026HEARTS: Benchmarking LLM Reasoning on Health Time Seriescited by 8
  12. 2026TeamBench: Evaluating Agent Coordination under Enforced Role Separationcited by 8
38 more
  1. 2026On the Reliability Limits of LLM-Based Multi-Agent Planningcited by 7
  2. 2026Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarkingcited by 7
  3. 2026AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentationcited by 7
  4. 2026Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptationcited by 7
  5. 2026Proof of Time: A Benchmark for Evaluating Scientific Idea Judgmentscited by 5
  6. 2026Table-as-Search: Formulate Long-Horizon Agentic Information Seeking as Table Completioncited by 5
  7. 2026Sovereign Agentic Loops: Decoupling AI Reasoning from Execution in Real-World Systemscited by 5
  8. 2026Token Economics for LLM Agents: A Dual-View Study from Computing and Economicscited by 5
  9. 2026Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systemscited by 5
  10. 2026EvoMAS: Evolutionary Generation of Multi-Agent Systemscited by 4
  11. 2026El Agente Gr'afico: A Semantic Execution Runtime for Scientific Agentscited by 4
  12. 2026DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensemblescited by 4
  13. 2026Holos: A Web-Scale LLM-Based Multi-Agent System for the Agentic Webcited by 4
  14. 2026TacoMAS: Test-Time Co-Evolution of Topology and Capability in LLM-based Multi-Agent Systemscited by 4
  15. 2026The Illusion of Multi-Agent Advantagecited by 4
  16. 2026MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?cited by 4
  17. 2026AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systemscited by 3
  18. 2026Phase Transition for Budgeted Multi-Agent Synergycited by 3
  19. 2026When Does Multi-Agent Collaboration Help? An Entropy Perspectivecited by 3
  20. 2026Don't Trust Stubborn Neighbors: A Security Framework for Agentic Networkscited by 3
  21. 2026Scaling Teams or Scaling Time? Memory Enabled Lifelong Learning in LLM Multi-Agent Systemscited by 3
  22. 2026AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Taskscited by 3
  23. 2026The Ringelmann Effect in Multi-Agent LLM Systems: A Scaling Law for Effective Team Sizecited by 3
  24. 2026Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactionscited by 3
  25. 2026Scaling Multiagent Systems with Process Rewardscited by 2
  26. 2026ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulationcited by 2
  27. 2026How Well Do Multimodal Models Reason on ECG Signals?cited by 2
  28. 2026Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspacescited by 2
  29. 2026SPD-RAG: Sub-Agent Per Document Retrieval-Augmented Generationcited by 2
  30. 2026Do Agent Societies Develop Intellectual Elites? The Hidden Power Laws of Collective Cognition in LLM Multi-Agent Systemscited by 2
  31. 2026EMS: Multi-Agent Voting via Efficient Majority-then-Stoppingcited by 2
  32. 2026HCP-MAD:Heterogeneous Consensus-Progressive Reasoning for Efficient Multi-Agent Debatecited by 2
  33. 2026In-Context Prompting Obsoletes Agent Orchestration for Procedural Taskscited by 2
  34. 2026Improving the Efficiency of Language Agent Teams with Adaptive Task Graphscited by 2
  35. 2026QUIVER: A Formal Framework for Quantifying Perturbation Propagation and Bifurcation in Compound AI Systemscited by 2
  36. 2026Exploring the Topology and Memory of Consensus: How LLM Agents Agree, Fragment, or Settle When Forming Conventionscited by 2
  37. 2026Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaborationcited by 2
  38. 2026State-dependent error correlations shape voting thresholds in committees of AI agentscited by 2

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
  2. 2026Scaling Behavior of Single LLM-Driven Multi-Agent Systems
  3. 2025Single-Agent Scaling Fails Multi-Agent Intelligence: Towards Foundation Models with Native Multi-Agent Intelligence
  4. 2025AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprise
  5. 2026Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scaling
  6. 2026Apodex 1.1: Scaling Agentic Intelligence for Complex Work
  7. 2026Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability Analysis
  8. 2026Can Small Agents Collaborate to Beat a Single Large Language Model?