Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents

Agents that synthesize their own tools ship a second artifact alongside each answer: a software library that future tasks reuse, compose, and depend on. Task completion (TC) certifies the answer; it does not certify the library. On a Claude Haiku 4.5 pilot, we patch the harness to preserve per-tool source and replay every synthesised tool against a held-out conformance suite. Across 222 preserved tools and three protocols, 96.8% record per-tool correctness C=0.00: two protocols silent-rot at 100%, one at 91.7%. Hand-written reference implementations score C=1.00 on all 16 capability suites. The synthesised tools execute cleanly and the in-session verifier raises no flag; on held-out inputs they simply return wrong answers. This is the verification-vs.-conformance gap a pass-rate benchmark cannot see. EvolveTool-Bench makes that gap measurable. Sessions are structured into seed, gap, variant, composition, regression, and adversarial roles; runs emit per-tool manifests, verified-subset TC, correct-vs.-incorrect reuse, and audit traces for post-deployment monitoring. In the Haiku pilot over five protocols (3 seeds, 8 sessions, 51 verified decisions per pass), TC alone does not separate the protocols; the audit layer does. One pre-specified contrast shows the adapted decision-split synthesis protocol underperforming one-shot synthesis (-7.1 pp, 95% CI [-13.9,-0.1]; not significant after BH correction) -- a hypothesis-generating signal, not a confirmed effect. We release the harness, per-tool source preservation, conformance replay, and reproducibility manifests. An audit schema of this shape -- verifier coverage, reuse decomposition, held-out conformance -- is the minimum a deployed tool-evolving agent should emit between releases.