What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead
Not on the map. Why?
Abstract (arXiv)
Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, popularity lists, hand-curated frames, or pipelines that repair a server until it starts. We report what an unrepaired probability sample actually contains. From a 24,135-server registry census we draw 400 npm/stdio servers with a published seed and probe each one over the wire. Only 48.8% complete an initialize handshake, against 66.7% for a hand-curated frame measured with the same instrument, and the dominant failure is not missing credentials (13.3%) but servers that never start at all (37.5%). Among the 195 that do run, hard conformance is total: zero fatal JSON Schema violations across 2,766 advertised tools. Optional safety annotations are the real variance, and the tool-level omission rate on a random draw is 58.8% against 41.5% on the curated frame, so curation flatters this figure too. We then compare the tool descriptions these servers advertise against two tool-use benchmark corpora using one method held constant. Real MCP tools show 2.8% near-duplication at cosine 0.70, and all of it lies within single servers: cross-author near-duplication is 0.0% at every threshold tested. BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks. UltraTool shows 0.3%, cleaner than real tools, so this is a property of BFCL and not of synthetic corpora as a class. Separately, 68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact name-plus-description repeats, against 0.4% for real MCP, so any statistic computed over these releases without global deduplication measures repetition rather than tools. All figures regenerate from released scripts and a published seed.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 6 of its 10 references that are in the map. First posted on arXiv 10 Sep 2026.
Builds on
6 of 10 references.
References · 6
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2026Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluationcited by 8
- 2026Evaluating Tool Cloning in Agentic-AI Ecosystemscited by 2
- 2026Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliabilitycited by 2
- 2026Registry Descriptions Go Stale Unevenly: An 89-Day Measurement of Model Context Protocol Drift, and Why Drift-Ranked Re-Auditing Under-Covers Itcited by 2
- 2026Exposed by Design: A Dynamic Security Assessment of Internet-Facing MCP Servers at Scalecited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Same Name, Different Server: A Security Census of Silent Drift in the Model Context Protocol Ecosystem
- 2026A Taxonomy of Runtime Faults in Model Context Protocol Servers
- 2026MCP-in-SoS: Risk assessment framework for open-source MCP servers
- 2025A Measurement Study of Model Context Protocol Ecosystem
- 2026Description-Code Inconsistency in Real-world MCP Servers: Measurement, Detection, and Security Implications
- 2026An Empirical Study of Model Context Protocol Applications
- 2026Unsafe by Flow: Uncovering Bidirectional Data-Flow Risks in MCP Ecosystem
- 2026A Large-Scale Dataset of MCP Implementations on GitHub