mlsanityData to 14 Oct 2026

MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

arXiv 2508.20453 ↗First posted 28 Aug 2025cs.CLcited by 109

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

We introduce MCP-Bench, a benchmark for evaluating large language models (LLMs) on realistic, multi-step tasks that demand tool use, cross-tool coordination, precise parameter control, and planning/reasoning for solving tasks. Built on the Model Context Protocol (MCP), MCP-Bench connects LLMs to 28 representative live MCP servers spanning 250 tools across domains such as finance, traveling, scientific computing, and academic search. Unlike prior API-based benchmarks, each MCP server provides a set of complementary tools designed to work together, enabling the construction of authentic, multi-step tasks with rich input-output coupling. Tasks in MCP-Bench test agents' ability to retrieve relevant tools from fuzzy instructions without explicit tool names, plan multi-hop execution trajectories for complex objectives, ground responses in intermediate tool outputs, and orchestrate cross-domain workflows - capabilities not adequately evaluated by existing benchmarks that rely on explicit tool specifications, shallow few-step workflows, and isolated domain operations. We propose a multi-faceted evaluation framework covering tool-level schema understanding and usage, trajectory-level planning, and task completion. Experiments on 20 advanced LLMs reveal persistent challenges in MCP-Bench. Code and data: https://github.com/Accenture/mcp-bench.

Key facts. Cited by 109 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 20 of its 48 references that are in the map. First posted on arXiv 28 Aug 2025.

Builds on

20 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models17 Jul
  2. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models22 May
  3. 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario17 Jan
  4. 2024ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMs20 Nov

Other references · 16

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  5. 2025Kimi K2: Open Agentic Intelligencecited by 567
  6. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  7. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  8. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  9. 2024Capabilities of Gemini Models in Medicinecited by 237
  10. 2024TradingAgents: Multi-Agents LLM Financial Trading Frameworkcited by 188
  11. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 47
  12. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
4 more
  1. 2025MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agentscited by 38
  2. 2025REALM-Bench: A Benchmark for Evaluating Multi-Agent Systems on Real-world, Dynamic Planning and Scheduling Taskscited by 16
  3. 2025ActionStudio: A Lightweight Framework for Data and Training of Large Action Modelscited by 6
  4. 2025C3-Bench: The Things Real Disturbing LLM based Agent in Multi-Taskingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 92.

109 papers.

Sort:

By area:

  1. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 69
  2. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 55
  3. 2026WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluationcited by 37
  4. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 33
  5. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 33
  6. 2026ClawBench: Can AI Agents Complete Everyday Online Tasks?cited by 33
  7. 2025ARE: Scaling Up Agent Environments and Evaluationscited by 30
  8. 2025OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agentscited by 29
  9. 2026Benchmark Test-Time Scaling of General LLM Agentscited by 26
  10. 2026From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agentscited by 24
  11. 2026PhysicianBench: Evaluating LLM Agents in Real-World EHR Environmentscited by 21
  12. 2026Breaking the Protocol: Security Analysis of the Model Context Protocol Specification and Prompt Injection Vulnerabilities in Tool-Integrated LLM Agentscited by 19
38 more
  1. 2026ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agentscited by 17
  2. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 16
  3. 2026Learning Agent-Compatible Context Management for Long-Horizon Taskscited by 15
  4. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  5. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  6. 2026Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluationcited by 13
  7. 2026DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Usecited by 13
  8. 2026Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflowscited by 13
  9. 2025M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmarkcited by 11
  10. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  11. 2026World of Workflows: A Benchmark for Bringing World Models to Enterprise Systemscited by 11
  12. 2026PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystemscited by 11
  13. 2025Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agentscited by 10
  14. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 10
  15. 2025MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Usecited by 9
  16. 2026LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarioscited by 9
  17. 2026PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generationcited by 8
  18. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  19. 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulationcited by 7
  20. 2026AgentSkiller: Scaling Generalist Agent Intelligence through Semantically Integrated Cross-Domain Data Synthesiscited by 6
  21. 2026Red Skills or Blue Skills? A Dive Into Skills Published on ClawHubcited by 5
  22. 2026From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agentscited by 5
  23. 2026Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspacescited by 4
  24. 2026PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Toolscited by 4
  25. 2026FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Taskscited by 4
  26. 2026An AI Agent Execution Environment to Safeguard User Datacited by 4
  27. 2026ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandboxcited by 4
  28. 2026STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamicscited by 4
  29. 2026SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agentscited by 4
  30. 2026Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Usecited by 4
  31. 2025AgentFlux: Decoupled Fine-Tuning & Inference for On-Device Agentic Systemscited by 3
  32. 2025InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agentscited by 3
  33. 2025Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Frameworkcited by 3
  34. 2025Benchmark for Planning and Control with Large Language Model Agents: Blocksworld with Model Context Protocolcited by 3
  35. 2026C-World: A Computer Use Agent Environment Creatorcited by 3
  36. 2026MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integrationcited by 3
  37. 2026From Tool Orchestration to Code Execution: A Study of MCP Design Choicescited by 3
  38. 2026Oblivion: Self-Adaptive Agentic Memory Control through Decay-Driven Activationcited by 3

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers
  2. 2025MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools
  3. 2025MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools
  4. 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
  5. 2025ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agents
  6. 2025MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use
  7. 2025LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?
  8. 2025Help or Hurdle? Rethinking Model Context Protocol-Augmented Large Language Models