TeamBench: Evaluating Agent Coordination under Enforced Role Separation
Not on the map. Why?
Abstract (arXiv)
Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls. Without enforcement, a team pass rate can mask whether agents actually coordinated or whether one role effectively did another role's work. We present TeamBench, a benchmark with 851 task templates and 931 seeded instances for evaluating agent coordination under operating system-enforced role separation. TeamBench separates specification access, workspace editing, and final certification across Planner, Executor, and Verifier roles, so that no role can read the full requirements, modify the workspace, and certify the final answer. Prompt-only and sandbox-enforced teams reach statistically indistinguishable pass rates, but prompt-only runs produce 3.6 times more cases where the verifier attempts to edit the executor's code. Verifiers approve 49% of submissions that fail the deterministic grader, and removing the verifier improves mean partial score in the ablation. Team value is also conditional. Teams benefit when single agents struggle, but hurt when single agents already perform well. A 40-session human study under the same role separation shows that our benchmark exposes interaction patterns that pass rate misses. Solo participants work through the task directly, human participants paired with agents often collapse into quick approval, and human teams spend more effort coordinating missing information across roles.
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 20 of its 34 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
20 of 34 references.
References · 20
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2018Datasheets for Datasetscited by 1,371
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
- 2023ChatDev: Communicative Agents for Software Developmentcited by 893
8 more
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
- 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
- 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
- 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 169
- 2025Towards a Science of Scaling Agent Systemscited by 125
- 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
- 2024Language Agents as Optimizable Graphscited by 81
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026PerspectiveGap: A Benchmark for Multi-Agent Orchestration Promptingcited by 6
- 2026MatrAIx: Simulating the World with 8.3 Billion Persona Agentscited by 5
- 2026A Large-Scale Benchmark and Risk Assessment of Traffic Analysis Attacks on Cloud LLM Servicescited by 1
- 2026Where Knowledge and Authority Sit Changes What an Agent Benchmark Can Resolvecited by 0
- 2026AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineeringcited by 0
- 2026Theory of Scene: Breaking the Symmetry Trap in Multi-Agent LLM Coordinationcited by 0
- 2026OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtimecited by 0
- 2026AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelinescited by 0
- 2026AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines1 Oct 2026
- 2026OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime29 Sep 2026
- 2026Theory of Scene: Breaking the Symmetry Trap in Multi-Agent LLM Coordination26 Sep 2026
- 2026AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering26 Sep 2026
- 2026A Large-Scale Benchmark and Risk Assessment of Traffic Analysis Attacks on Cloud LLM Services25 Sep 2026
- 2026MatrAIx: Simulating the World with 8.3 Billion Persona Agents4 Aug 2026
- 2026Where Knowledge and Authority Sit Changes What an Agent Benchmark Can Resolve3 Jul 2026
- 2026PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting7 Jun 2026