# MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use arXiv: https://arxiv.org/abs/2508.16260 · HTML: https://mlsanity.com/paper/2508.16260/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. - Builds on 21 of its 35 references that are in the map. - First posted on arXiv 22 Aug 2025. ## Placed in - LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/) ## Builds on (most cited first, 21 of 21) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023) - [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023) - [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023) - [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023) - [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023) - [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023) - [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024) - [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://mlsanity.com/paper/2305.11554/) (2023) - [OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation](https://mlsanity.com/paper/2505.23885/) (2025) - [Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement](https://mlsanity.com/paper/2406.11176/) (2024) - [ACEBench: Who Wins the Match Point in Tool Usage?](https://mlsanity.com/paper/2501.12851/) (2025) - [ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario](https://mlsanity.com/paper/2501.10132/) (2025) - [MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models](https://mlsanity.com/paper/2505.16700/) (2025) - [Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments](https://mlsanity.com/paper/2402.14672/) (2024) - [Efficient and Scalable Estimation of Tool Representations in Vector Space](https://mlsanity.com/paper/2409.02141/) (2024) - [MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation](https://mlsanity.com/paper/2407.12871/) (2024) - [AutoFeedback: An LLM-based Framework for Efficient and Accurate API Request Generation](https://mlsanity.com/paper/2410.06943/) (2024) ## Built on by (most cited first, 16 of 16) - [MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers](https://mlsanity.com/paper/2602.00933/) (2026) - [ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces](https://mlsanity.com/paper/2604.05172/) (2026) - [AgentAbstain: Do LLM Agents Know When Not to Act?](https://mlsanity.com/paper/2607.10059/) (2026) - [Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents](https://mlsanity.com/paper/2510.04491/) (2025) - [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026) - [TheMCPCompany: Creating General-purpose Agents with Task-specific Tools](https://mlsanity.com/paper/2510.19286/) (2025) - [MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration](https://mlsanity.com/paper/2601.23049/) (2026) - [CUBE: A Standard for Unifying Agent Benchmarks](https://mlsanity.com/paper/2603.15798/) (2026) - [Overwhelmed by Choice: Studying LLM Decision Making at Scale](https://mlsanity.com/paper/2609.32809/) (2026) - [How Brittle is Agent Safety? Rethinking Agent Risk under Intent Concealment and Task Complexity](https://mlsanity.com/paper/2511.08487/) (2025) - [A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks](https://mlsanity.com/paper/2605.28556/) (2026) - [ToolAlignBench: Investigating Alignment Conflicts in Tool-Calling Enabled LLMs](https://mlsanity.com/paper/2607.14285/) (2026) - [DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers](https://mlsanity.com/paper/2607.20531/) (2026) - [Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance](https://mlsanity.com/paper/2608.03403/) (2026) - [Agent Seer: Synthesizing Scenarios from Specification Understanding](https://mlsanity.com/paper/2608.26133/) (2026) - [Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks](https://mlsanity.com/paper/2609.08404/) (2026) ## Abstract (arXiv) Large Language Models (LLMs) are evolving from text generators into reasoning agents. This transition makes their ability to use external tools a critical capability. However, evaluating this skill presents a significant challenge. Existing benchmarks are often limited by their reliance on synthetic tools and severely constrained action spaces. To address these limitations, we introduce MCPVerse, an expansive, real-world benchmark for evaluating agentic tool use. MCPVerse integrates more than 550 real-world, executable tools to create an unprecedented action space exceeding 140k tokens, and employs outcome-based evaluation with real-time ground truth for time-sensitive tasks. We benchmarked the state-of-the-art LLMs across three modes (Oracle, Standard, and Max-Scale), revealing that while most models suffer performance degradation when confronted with larger tool sets, the agentic models, such as Claude-4-Sonnet, can effectively leverage expanded exploration spaces to improve accuracy. This finding not only exposes the limitations of state-of-the-art models in complex, real-world scenarios but also establishes MCPVerse as a critical benchmark for measuring and advancing agentic tool use capabilities.