mlsanityData to 7 Oct 2026

Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems

arXiv 2603.01045 ↗First posted 1 Mar 2026cs.MA cs.AIcited by 16

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models are increasingly deployed in multi-agent systems to overcome context limitations by distributing information across agents. Yet whether agents can reliably compute with distributed information, rather than merely exchange it, remains an open question. We introduce SILO-BENCH, a role-agnostic benchmark of 30 algorithmic tasks across three communication complexity levels, evaluating 54 configurations over 1,620 experiments. Our experiments expose a fundamental Communication-Reasoning Gap: agents spontaneously form task-appropriate coordination topologies and exchange information actively, yet systematically fail to synthesize distributed state into correct answers. The failure is localized to the reasoning-integration stage where agents often acquire sufficient information but cannot integrate it. This coordination overhead compounds with scale, eventually eliminating parallelization gains entirely. These findings demonstrate that naively scaling agent count cannot circumvent context limitations, and SILO-BENCH provides a foundation for tracking progress toward genuinely collaborative multi-agent systems. The code is available at https://github.com/jwyjohn/acl26-silo-bench .

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 29 of its 44 references that are in the map. First posted on arXiv 1 Mar 2026.

Builds on

29 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Mixture-of-Agents Enhances Large Language Model Capabilities7 Jun
  2. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents9 Apr
  3. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?28 Feb
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar

Other references · 25

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  8. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  9. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  10. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  11. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  12. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
13 more
  1. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  2. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  3. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  4. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  5. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplaycited by 62
  6. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
  7. 2024M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitionscited by 38
  8. 2025Difficulty-Aware Agentic Orchestration for Query-Specific Multi-Agent Workflowscited by 23
  9. 2024An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Makingcited by 14
  10. 2025Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challengescited by 14
  11. 2025Talk2Image: A Multi-Agent System for Multi-Turn Image Generation and Editingcited by 13
  12. 2025MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Toolscited by 9
  13. 2026Towards Self-Robust LLMs: Intrinsic Prompt Noise Resistance via CoIPOcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026PerspectiveGap: A Benchmark for Multi-Agent Orchestration Promptingcited by 6
  2. 2026QueenBee Planner: Skill-Evolving Communication Topologies for Token-Efficient LLM Multi-Agent Systemscited by 3
  3. 2026Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflowscited by 1
  4. 2026Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflowscited by 1
  5. 2026Position: Multi-Agent Systems Should Prioritize Concurrency Controlcited by 1
  6. 2026CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologiescited by 1
  7. 2026FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systemscited by 0
  8. 2026Where Knowledge and Authority Sit Changes What an Agent Benchmark Can Resolvecited by 0
  9. 2026OrchBench: Evaluating Multi-Agent Orchestration Plans in Isolation via Deterministic Simulationcited by 0
  10. 2026ToMAS: A Pilot Failure-Grounded Theory-of-Mind Benchmark from Multi-Agent LLM Failurescited by 0
  11. 2026A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systemscited by 0
  12. 2026Reinforcement Learning of Communication in a Mesh of Small Language Modelscited by 0
4 more
  1. 2026RepoMAS: Solving Progressively Specified Tasks with Issue-Driven Multi-Agent Systemscited by 0
  2. 2026AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineeringcited by 0
  3. 2026Multi-agent discussion gains less when dissent is withheldcited by 0
  4. 2026Pay for the Fault, Not the Flow: Label-Free In-Flow Multi-Agent Workflow Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability
  2. 2026DPBench: Structural Determinants of Multi-Agent LLM Coordination Under Simultaneous Resource Contention
  3. 2025AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMs
  4. 2026Scaling Behavior of Single LLM-Driven Multi-Agent Systems
  5. 2026Decentralized Multi-Agent Systems with Shared Context
  6. 2025Benchmarking LLMs' Swarm intelligence
  7. 2025Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs
  8. 2026Benchmarking Emergent Coordination in Large-Scale LLM Populations: An Evaluation Framework on the MoltBook Archive