mlsanityData to 14 Oct 2026

What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead

arXiv 2609.10962 ↗First posted 10 Sep 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, popularity lists, hand-curated frames, or pipelines that repair a server until it starts. We report what an unrepaired probability sample actually contains. From a 24,135-server registry census we draw 400 npm/stdio servers with a published seed and probe each one over the wire. Only 48.8% complete an initialize handshake, against 66.7% for a hand-curated frame measured with the same instrument, and the dominant failure is not missing credentials (13.3%) but servers that never start at all (37.5%). Among the 195 that do run, hard conformance is total: zero fatal JSON Schema violations across 2,766 advertised tools. Optional safety annotations are the real variance, and the tool-level omission rate on a random draw is 58.8% against 41.5% on the curated frame, so curation flatters this figure too. We then compare the tool descriptions these servers advertise against two tool-use benchmark corpora using one method held constant. Real MCP tools show 2.8% near-duplication at cosine 0.70, and all of it lies within single servers: cross-author near-duplication is 0.0% at every threshold tested. BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks. UltraTool shows 0.3%, cleaner than real tools, so this is a property of BFCL and not of synthetic corpora as a class. Separately, 68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact name-plus-description repeats, against 0.4% for real MCP, so any statistic computed over these releases without global deduplication measures repetition rather than tools. All figures regenerate from released scripts and a published seed.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 6 of its 10 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

6 of 10 references.

References · 6

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  2. 2026Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluationcited by 8
  3. 2026Evaluating Tool Cloning in Agentic-AI Ecosystemscited by 2
  4. 2026Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliabilitycited by 2
  5. 2026Registry Descriptions Go Stale Unevenly: An 89-Day Measurement of Model Context Protocol Drift, and Why Drift-Ranked Re-Auditing Under-Covers Itcited by 2
  6. 2026Exposed by Design: A Dynamic Security Assessment of Internet-Facing MCP Servers at Scalecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Same Name, Different Server: A Security Census of Silent Drift in the Model Context Protocol Ecosystem
  2. 2026A Taxonomy of Runtime Faults in Model Context Protocol Servers
  3. 2026MCP-in-SoS: Risk assessment framework for open-source MCP servers
  4. 2025A Measurement Study of Model Context Protocol Ecosystem
  5. 2026Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications
  6. 2026An Empirical Study of Model Context Protocol Applications
  7. 2026Unsafe by Flow: Uncovering Bidirectional Data-Flow Risks in MCP Ecosystem
  8. 2026A Large-Scale Dataset of MCP Implementations on GitHub