mlsanityData to 14 Oct 2026

MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models

arXiv 2507.12806 ↗First posted 17 Jul 2025cs.AI cs.CLcited by 36

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

The rapid rise of Large Language Models (LLMs)-based intelligent agents underscores the need for robust, scalable evaluation frameworks. Existing methods rely on static benchmarks and labor-intensive data collection, limiting practical assessment. We introduce MCPEval, an open-source Model Context Protocol (MCP)-based framework that automates end-to-end task generation and deep evaluation of LLM agents across diverse domains. MCPEval standardizes metrics, seamlessly integrates with native agent tools, and eliminates manual effort in building evaluation pipelines. Empirical results across five real-world domains show its effectiveness in revealing nuanced, domain-specific performance. We publicly release MCPEval https://github.com/SalesforceAIResearch/MCPEval to promote reproducible and standardized LLM agent evaluation.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 16 of its 32 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

16 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agents9 May
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul

Other references · 14

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  7. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  8. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 84
  9. 2025MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agentscited by 38
  10. 2025PersonaBench: Evaluating AI Models on Understanding Personal Information through Accessing (Synthetic) Private User Datacited by 27
  11. 2023DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AIcited by 22
  12. 2024PRACT: Optimizing Principled Reasoning and Acting of LLM Agentcited by 6
2 more
  1. 2025ActionStudio: A Lightweight Framework for Data and Training of Large Action Modelscited by 6
  2. 2025LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedbackcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 21.

36 papers.

Sort:

By area:

  1. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Serverscited by 109
  2. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
  3. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 85
  4. 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Serverscited by 53
  5. 2025LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?cited by 48
  6. 2025MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Usecited by 39
  7. 2025MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Toolscited by 32
  8. 2026SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?cited by 30
  9. 2025OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agentscited by 29
  10. 2026From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agentscited by 24
  11. 2025RAFFLES: Reasoning-based Attribution of Faults for LLM Systemscited by 21
  12. 2025LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queriescited by 17
24 more
  1. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptionscited by 17
  2. 2025MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Toolscited by 11
  3. 2025M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmarkcited by 11
  4. 2025Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agentscited by 10
  5. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 10
  6. 2025TheMCPCompany: Creating General-purpose Agents with Task-specific Toolscited by 4
  7. 2026ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandboxcited by 4
  8. 2026Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Usecited by 4
  9. 2026Polymer-Agent: Large Language Model Agent for Polymer Designcited by 3
  10. 2026MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integrationcited by 3
  11. 2026MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulationcited by 3
  12. 2026User-Oriented Multi-Turn Dialogue Generation with Tool Use at scalecited by 2
  13. 2026From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agentscited by 2
  14. 2026E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarioscited by 2
  15. 2026GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Taskscited by 2
  16. 2025Can LLM Infer Risk Information From MCP Server System Logs?cited by 1
  17. 2026EnterpriseLab: A Full-Stack Platform for developing and deploying agents in Enterprisescited by 1
  18. 2025ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystemcited by 0
  19. 2026Towards a Declarative Agentic Layer for Intelligent Agents in MCP-Based Server Ecosystemscited by 0
  20. 2026MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Serverscited by 0
  21. 2026Cross-View Correspondence Is a Measurement Intervention: Two-Sided Validation for Agent Evaluation and Credit Assignmentcited by 0
  22. 2026SkillAtlas: An Attack Trace Library for Agent Skillscited by 0
  23. 2026Beyond Perfect Teammates: Multi-Agent Collaboration with Unreliable Colleaguescited by 0
  24. 2026AgentStripes: An Evidence-Graded Maturity Model for AI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use
  2. 2026DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers
  3. 2025MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
  4. 2026The Evaluation Context Protocol (ECP): A Portable Contract for AI Agent Evaluation
  5. 2025MCPSecBench: A Systematic Security Benchmark and Playground for Testing Model Context Protocols
  6. 2026AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
  7. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  8. 2025Help or Hurdle? Rethinking Model Context Protocol-Augmented Large Language Models