mlsanityData to 7 Oct 2026

MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

arXiv 2602.00933 ↗First posted 31 Jan 2026cs.SE cs.AIcited by 46

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evaluations fall short along three key axes: realistic multi-step workflows with cross-server orchestration, breadth across authentic MCP servers rather than mocks, and structured, reproducible claim-level scoring disentangled from agent verbosity or style. We introduce MCP-Atlas, a benchmark for measuring tool-use competency against production MCP servers. MCP-Atlas contains 1,000 natural-language tasks written and verified by human experts spanning 36 real MCP servers and 220 tools. Prompts do not specify servers, tools, or parameters, requiring agents to identify relevant tools among semantically plausible distractors and to compose multi-step, cross-server workflows. Each task is scored with a claim-level rubric, where final answers are scored against atomic factual claims grounded in tool outputs. This answer-centric scoring permits valid alternative tool-call trajectories to receive credit. We pair this with an 11-category diagnostic taxonomy that disentangles tool-call failures from cognitive failures in task understanding, synthesis, parsing, and stopping. Evaluating 20 frontier models from six providers under matched task-level conditions, we find pass rates up to 82.2% at a 0.75 claim coverage threshold and a clear three-tier performance structure. Automated diagnostics show that 63.3% of diagnosed failures are cognitive rather than tool-call related. Notably, several high-performing models fail after successful tool execution due to premature stopping or incorrect synthesis. We release the task schema, containerized harness, claim evaluator, and a 500-task public split, while reserving a 500-task private split to preserve leaderboard integrity. The code is at https://github.com/scaleapi/mcp-atlas.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 22 of its 46 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

22 of 46 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use22 Aug
  2. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers20 Aug
  3. 2025LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?3 Aug
  4. 2025MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models17 Jul
  5. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models22 May
  6. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios1 Jan
  7. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul

Other references · 14

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  7. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  10. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  11. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  12. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 73
2 more
  1. 2025MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Usecited by 37
  2. 2024Why am I Still Seeing This: Measuring the Effectiveness Of Ad Controls and Explanations in AI-Mediated Ad Targeting Systemscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 46.

46 papers.

Sort:

By area:

  1. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  2. 2026Kimi K3: Open Frontier Intelligencecited by 99
  3. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  4. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  5. 2026SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?cited by 28
  6. 2026Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agentscited by 28
  7. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
  8. 2026HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?cited by 16
  9. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  10. 2026EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RLcited by 11
  11. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  12. 2026Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Modelcited by 9
34 more
  1. 2026Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptationcited by 7
  2. 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulationcited by 6
  3. 2026CocoaBench: Evaluating Unified Digital Agents in the Wildcited by 6
  4. 2026LHAW: Controllable Underspecification for Long-Horizon Taskscited by 5
  5. 2026Tracing Agentic Failure from the Flow of Successcited by 5
  6. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systemscited by 5
  7. 2026From Tool Orchestration to Code Execution: A Study of MCP Design Choicescited by 3
  8. 2026Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Designcited by 3
  9. 2026Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?cited by 2
  10. 2026ADK Arena: Evaluating Agent Development Kits via LLM-as-a-Developercited by 2
  11. 2026MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agentscited by 2
  12. 2026Solar Open 2 Technical Reportcited by 2
  13. 2026AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?cited by 2
  14. 2026ClawGym II: Exploring Black-Box RL on Agent Harnesscited by 2
  15. 2026One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflowscited by 2
  16. 2026PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agentscited by 2
  17. 2026ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agentscited by 1
  18. 2026OmniaBench: Benchmarking General AI Agents Across Diverse Scenarioscited by 1
  19. 2026E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarioscited by 1
  20. 2026LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexingcited by 1
  21. 2026ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Stepcited by 1
  22. 2026Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Toolscited by 1
  23. 2026Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMscited by 0
  24. 2026Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branchescited by 0
  25. 2026SwarmX: Agentic Scheduling for Low-Latency Agentic Systemscited by 0
  26. 2026Forethought: Verifiable Reasoning from Neurosymbolic Primitive Programmingcited by 0
  27. 2026Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utilitycited by 0
  28. 2026DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Serverscited by 0
  29. 2026AgentMercury: Your Agent Can Synthesize Verifiable Environments for Business Scenarios at scalecited by 0
  30. 2026XYEval: Agents say yes to bad advicecited by 0
  31. 2026Rufus-Air: An Open LLM Post-Training Recipecited by 0
  32. 2026Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agentscited by 0
  33. 2026Who Holds the Pen? Let Specifications, Not Agents, Sign Offcited by 0
  34. 2026Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
  2. 2025MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
  3. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions
  4. 2025MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools
  5. 2025LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
  6. 2025ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agents
  7. 2026ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox
  8. 2025MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools