# ProtocolBench: Which LLM MultiAgent Protocol to Choose?

arXiv: https://arxiv.org/abs/2510.17149 · HTML: https://mlsanity.com/paper/2510.17149/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://mlsanity.com/paper/2503.01935/) (2025)
- [A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP)](https://mlsanity.com/paper/2505.02279/) (2025)
- [Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence](https://mlsanity.com/paper/2407.07061/) (2024)
- [A Survey of AI Agent Protocols](https://mlsanity.com/paper/2504.16736/) (2025)
- [A Scalable Communication Protocol for Networks of Large Language Models](https://mlsanity.com/paper/2410.11905/) (2024)
- [CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale](https://mlsanity.com/paper/2507.05178/) (2025)

## Built on by (most cited first, 6 of 6)

- [Rethinking the Evaluation of Efficiency Methods for Multi-Agent Systems](https://mlsanity.com/paper/2609.05933/) (2026)
- [BusMA: A Bus Communication Substrate for Multi-Agent Systems](https://mlsanity.com/paper/2609.15054/) (2026)
- [Public-Sharing Labels and Verbatim Field Egress in an MCP-to-A2A Agent Configuration: A Controlled Multi-Model Study](https://mlsanity.com/paper/2609.01693/) (2026)
- [Relic: From Multi-Agent Collaboration to Persistent Organizational Capability](https://mlsanity.com/paper/2609.32965/) (2026)
- [The Cost of a Hop: Benchmarking NLIP and A2A](https://mlsanity.com/paper/2610.04053/) (2026)
- [Why Do Agent Teams Fail Across Boundaries? A Failure Taxonomy for Agent-to-Agent Systems with Private States](https://mlsanity.com/paper/2610.19342/) (2026)

## Abstract (arXiv)

As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability. Despite the existence of diverse protocols (A2A, ACP, ANP, Agora, etc.), selection is often intuition-driven and lacks standardized guidance. We introduce ProtocolBench, a benchmark that systematically compares agent protocols along four measurable axes: task success, end-to-end latency, message or byte overhead, and robustness under failures. On ProtocolBench, protocol choice significantly influences system behavior. In the Streaming Queue scenario, overall completion time varies by up to 36.5% across protocols, and mean end-to-end latency differs by 3.48 s. Under Fail-Storm Recovery, resilience also differs consistently across protocols. Beyond evaluation, we present ProtocolRouter, a learnable protocol router that selects per-scenario (or per-module) protocols from requirement and runtime signals. ProtocolRouter reduces Fail-Storm recovery time by up to 18.1% versus the best single-protocol baseline, and achieves scenario-specific gains such as higher success in GAIA. We also release ProtocolRouterBench to standardize protocol evaluation and improve reliability at scale.