mlsanityData to 14 Oct 2026

LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries

arXiv 2508.15760 ↗First posted 21 Aug 2025cs.CL cs.AIcited by 17

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Tool calling has emerged as a critical capability for AI agents. In contrast to conventional tool calling frameworks that rely on static, provider-specific tool definitions, the Model Context Protocol (MCP) offers a unified interface to discover and invoke tools dynamically. However, there is a significant gap in benchmarking multi-step tasks using diverse MCP tools in realistic, dynamic scenarios. In this work, we present LiveMCP-101, a benchmark of 101 real-world queries that require coordinated use of multiple MCP tools. To address temporal variability in real-world tool responses, we introduce a parallel evaluation framework where a reference agent executes a validated plan simultaneously to produce real-time reference outputs. Experiments show that even frontier LLMs achieve a success rate below 60%, highlighting challenges in multi-step tool use. Comprehensive error analysis identifies seven failure modes spanning tool planning, parameterization, and output handling, pointing to concrete directions for improving current models. LiveMCP-101 sets a rigorous standard for evaluating real-world agent capabilities, advancing toward autonomous agent systems that reliably execute complex tasks through MCP tool orchestration.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 40 of its 54 references that are in the map. First posted on arXiv 21 Aug 2025.

Builds on

40 of 54 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers20 Aug
  2. 2025LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?3 Aug
  3. 2025MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models17 Jul
  4. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models22 May
  5. 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario17 Jan
  6. 2024Tool Learning with Large Language Models: A Survey28 May
  7. 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark14 May
  8. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models12 Mar
  9. 2024Learning to Use Tools via Cooperative and Interactive Agents5 Mar
  10. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls6 Feb
  11. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction11 Jan
  12. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback19 Sep
6 more
  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  3. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  4. 2023Tool Learning with Foundation Models17 Apr
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 22

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  8. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  9. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  10. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  11. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 265
  12. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 185
10 more
  1. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
  2. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 105
  3. 2025A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP)cited by 101
  4. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 82
  5. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  6. 2024ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrievalcited by 41
  7. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 40
  8. 2025WebRollback: Enhancing Web Agents with Explicit Rollback Mechanismscited by 15
  9. 2025Evaluation Report on MCP Serverscited by 9
  10. 2025LogicIF: Towards Complex Logic Instruction Followingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 85
  2. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 69
  3. 2025MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Usecited by 39
  4. 2025MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Toolscited by 11
  5. 2025Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agentscited by 10
  6. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 10
  7. 2026Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomycited by 6
  8. 2025SynthTools: A Framework for Scaling Synthetic Tools for Agent Developmentcited by 5
  9. 2025TheMCPCompany: Creating General-purpose Agents with Task-specific Toolscited by 4
  10. 2025Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generationcited by 3
  11. 2026MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulationcited by 3
  12. 2025Communication to Completion: Modeling Collaborative Workflows with Intelligent Multi-Agent Communicationcited by 1
5 more
  1. 2026EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agentscited by 1
  2. 2026Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latencycited by 1
  3. 2025ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystemcited by 0
  4. 2026DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Serverscited by 0
  5. 2026Sensitivity or Criterion? Measuring and Moving When Language Models Actcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools
  2. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
  3. 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
  4. 2026ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox
  5. 2025MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
  6. 2026MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers
  7. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions
  8. 2025MCP-Zero: Active Tool Discovery for Autonomous LLM Agents