# MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools

arXiv: https://arxiv.org/abs/2509.09734 · HTML: https://mlsanity.com/paper/2509.09734/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions](https://mlsanity.com/paper/2503.23278/) (2025)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [An Illusion of Progress? Assessing the Current State of Web Agents](https://mlsanity.com/paper/2504.01382/) (2025)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [ScreenAgent: A Vision Language Model-driven Computer Control Agent](https://mlsanity.com/paper/2402.07945/) (2024)
- [ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario](https://mlsanity.com/paper/2501.10132/) (2025)
- [MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models](https://mlsanity.com/paper/2505.16700/) (2025)
- [MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models](https://mlsanity.com/paper/2507.12806/) (2025)
- [Evaluation Report on MCP Servers](https://mlsanity.com/paper/2504.11094/) (2025)

## Built on by (most cited first, 30 of 32)

- [The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution](https://mlsanity.com/paper/2510.25726/) (2025)
- [TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments](https://mlsanity.com/paper/2510.01179/) (2025)
- [MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use](https://mlsanity.com/paper/2509.24002/) (2025)
- [SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?](https://mlsanity.com/paper/2603.00718/) (2026)
- [DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use](https://mlsanity.com/paper/2603.11076/) (2026)
- [Self-GC: Self-Governing Context for Long-Horizon LLM Agents](https://mlsanity.com/paper/2607.00692/) (2026)
- [DualTAP: A Dual-Task Adversarial Protector for Mobile MLLM Agents](https://mlsanity.com/paper/2511.13248/) (2025)
- [Quantifying Trust: Financial Risk Management for Trustworthy AI Agents](https://mlsanity.com/paper/2604.03976/) (2026)
- [ACE-Router: Generalizing History-Aware Routing from MCP Tools to the Agent Web](https://mlsanity.com/paper/2601.08276/) (2026)
- [FS-Researcher: Test-Time Scaling for Long-Horizon Research Tasks with File-System-Based Agents](https://mlsanity.com/paper/2602.01566/) (2026)
- [Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible](https://mlsanity.com/paper/2602.10139/) (2026)
- [Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy](https://mlsanity.com/paper/2603.05637/) (2026)
- [From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents](https://mlsanity.com/paper/2605.15104/) (2026)
- [Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation in an Uncertain Enterprise Environment](https://mlsanity.com/paper/2603.23638/) (2026)
- [Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents](https://mlsanity.com/paper/2605.08717/) (2026)
- [AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems](https://mlsanity.com/paper/2601.09393/) (2026)
- [MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration](https://mlsanity.com/paper/2601.23049/) (2026)
- [MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation](https://mlsanity.com/paper/2606.02470/) (2026)
- [AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments](https://mlsanity.com/paper/2607.05174/) (2026)
- [MCP-DPT: A Defense-Placement Taxonomy and Coverage Analysis for Model Context Protocol Security](https://mlsanity.com/paper/2604.07551/) (2026)
- [AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling](https://mlsanity.com/paper/2608.26623/) (2026)
- [Can LLM Infer Risk Information From MCP Server System Logs?](https://mlsanity.com/paper/2511.05867/) (2025)
- [Bypassing Document Ingestion: An MCP Approach to Financial Q&A](https://mlsanity.com/paper/2603.20316/) (2026)
- [HumanMCP: A Human-Like Query Dataset for Evaluating MCP Tool Retrieval Performance](https://mlsanity.com/paper/2602.23367/) (2025)
- [Making OpenAPI Documentation Agent-Ready: Detecting Documentation and REST Smells with a Multi-Agent LLM System](https://mlsanity.com/paper/2605.14312/) (2026)
- [AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculture](https://mlsanity.com/paper/2605.22366/) (2026)
- [Agentic Electronic Design Automation: A Handoff Perspective](https://mlsanity.com/paper/2606.19795/) (2026)
- [DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers](https://mlsanity.com/paper/2607.20531/) (2026)
- [Agent Seer: Synthesizing Scenarios from Specification Understanding](https://mlsanity.com/paper/2608.26133/) (2026)
- [ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents](https://mlsanity.com/paper/2609.12345/) (2026)

## Abstract (arXiv)

The Model Context Protocol (MCP) is rapidly emerging as a pivotal open standard, designed to enhance agent-tool integration and interoperability, and is positioned to unlock a new era of powerful, interconnected, and genuinely utilitarian agentic AI. However, despite MCP's growing adoption, existing benchmarks often fail to capture real-world agent performance within this new paradigm, leading to a distorted perception of their true operational value and an inability to reliably differentiate proficiencies. To bridge this critical evaluation gap, we introduce MCP-AgentBench -- a comprehensive benchmark specifically engineered to rigorously assess language agent capabilities in MCP-mediated tool interactions. Core contributions of MCP-AgentBench include: the establishment of a robust MCP testbed comprising 33 operational servers with 188 distinct tools; the development of a benchmark featuring 600 systematically designed queries distributed across 6 distinct categories of varying interaction complexity; and the introduction of MCP-Eval, a novel outcome-oriented evaluation methodology prioritizing real-world task success. Through extensive empirical evaluation of leading language agents, we provide foundational insights. MCP-AgentBench aims to equip the research community with a standardized and reliable framework to build, validate, and advance agents capable of fully leveraging MCP's transformative benefits, thereby accelerating progress toward truly capable and interoperable AI systems.