# MCPToolBench++: A Large Scale AI Agent Model Context Protocol MCP Tool Use Benchmark

arXiv: https://arxiv.org/abs/2508.07575 · HTML: https://mlsanity.com/paper/2508.07575/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 5 of 5)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models](https://mlsanity.com/paper/2505.16700/) (2025)
- [Evaluation Report on MCP Servers](https://mlsanity.com/paper/2504.11094/) (2025)

## Built on by (most cited first, 30 of 32)

- [Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning](https://mlsanity.com/paper/2602.10090/) (2026)
- [ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces](https://mlsanity.com/paper/2604.05172/) (2026)
- [Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions](https://mlsanity.com/paper/2602.14878/) (2026)
- [Beyond Max Tokens: Stealthy Resource Amplification via Tool Calling Chains in LLM Agents](https://mlsanity.com/paper/2601.10955/) (2026)
- [Environment Scaling for Interactive Agentic Experience Collection: A Survey](https://mlsanity.com/paper/2511.09586/) (2025)
- [OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability](https://mlsanity.com/paper/2511.20766/) (2025)
- [MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools](https://mlsanity.com/paper/2510.24284/) (2025)
- [World of Workflows: A Benchmark for Bringing World Models to Enterprise Systems](https://mlsanity.com/paper/2601.22130/) (2026)
- [MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use](https://mlsanity.com/paper/2512.24565/) (2025)
- [Beyond Task Completion: An Assessment Framework for Evaluating Agentic AI Systems](https://mlsanity.com/paper/2512.12791/) (2025)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [ACE-Router: Generalizing History-Aware Routing from MCP Tools to the Agent Web](https://mlsanity.com/paper/2601.08276/) (2026)
- [Automating Structural Engineering Workflows with Large Language Model Agents](https://mlsanity.com/paper/2510.11004/) (2025)
- [STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamics](https://mlsanity.com/paper/2605.18548/) (2026)
- [SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents](https://mlsanity.com/paper/2606.16591/) (2026)
- [MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation](https://mlsanity.com/paper/2606.02470/) (2026)
- [The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior](https://mlsanity.com/paper/2608.11386/) (2026)
- [PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents](https://mlsanity.com/paper/2608.24509/) (2026)
- [Bounded Autonomy for Enterprise AI: Typed Action Contracts and Consumer-Side Execution](https://mlsanity.com/paper/2604.14723/) (2026)
- [PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control](https://mlsanity.com/paper/2605.15963/) (2026)
- [Notation Matters: A Benchmark Study of Token-Optimized Formats in Agentic AI Systems](https://mlsanity.com/paper/2605.29676/) (2026)
- [Servant, Stalker, Predator: How An Honest, Helpful, And Harmless (3H) Agent Unlocks Adversarial Skills](https://mlsanity.com/paper/2508.19500/) (2025)
- [Learning to Recommend Multi-Agent Subgraphs from Calling Trees](https://mlsanity.com/paper/2601.22209/) (2026)
- [Private PoEtry: Private In-Context Learning via Product of Experts](https://mlsanity.com/paper/2602.05012/) (2026)
- [S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Services](https://mlsanity.com/paper/2602.14017/) (2026)
- [HumanMCP: A Human-Like Query Dataset for Evaluating MCP Tool Retrieval Performance](https://mlsanity.com/paper/2602.23367/) (2025)
- [STEM Agent: A Self-Adapting, Tool-Enabled, Extensible Architecture for Multi-Protocol AI Agent Systems](https://mlsanity.com/paper/2603.22359/) (2026)
- [TOBench: A Task-Oriented Omni-Modal Benchmark for Real-World Tool-Using Agents](https://mlsanity.com/paper/2605.16909/) (2026)
- [DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers](https://mlsanity.com/paper/2607.20531/) (2026)
- [Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends](https://mlsanity.com/paper/2609.33443/) (2026)

## Abstract (arXiv)

LLMs' capabilities are enhanced by using function calls to integrate various data sources or API results into the context window. Typical tools include search, web crawlers, maps, financial data, file systems, and browser usage, etc. Integrating these data sources or functions requires a standardized method. The Model Context Protocol (MCP) provides a standardized way to supply context to LLMs. However, the evaluation of LLMs and AI Agents' MCP tool use abilities suffer from several issues. First, there's a lack of comprehensive datasets or benchmarks to evaluate various MCP tools. Second, the diverse formats of response from MCP tool call execution further increase the difficulty of evaluation. Additionally, unlike existing tool-use benchmarks with high success rates in functions like programming and math functions, the success rate of real-world MCP tool is not guaranteed and varies across different MCP servers. Furthermore, the LLMs' context window also limits the number of available tools that can be called in a single run, because the textual descriptions of tool and the parameters have long token length for an LLM to process all at once. To help address the challenges of evaluating LLMs' performance on calling MCP tools, we propose MCPToolBench++, a large-scale, multi-domain AI Agent tool use benchmark. As of July 2025, this benchmark is build upon marketplace of over 4k MCP servers from more than 40 categories, collected from the MCP marketplaces and GitHub communities. The datasets consist of both single-step and multi-step tool calls across different categories. We evaluated SOTA LLMs with agentic abilities on this benchmark and reported the results.