Deployment-complete benchmarking
Not on the map. Why?
Abstract (arXiv)
Benchmarks increasingly guide deployment, procurement and scientific screening, yet a score supports only the response it records, not necessarily the deployment action. We introduce deployment-complete benchmarking, which tests whether benchmark evidence determines a deployment action. A benchmark is complete for a claim exactly when the action is constant on each evidence fiber; mixed fibers expose missing deployment information, and completion curves quantify the evidence required to resolve ambiguity. In controlled response spaces, benchmark-channel conformal coverage of 94.98% transferred poorly to an unmeasured deployment channel (10.07%), whereas response-rank intervals achieved 94.91% coverage; even zero benchmark error certified only 45.4% of candidates at the largest residual size. Public audits revealed incompleteness, including 97.9% mixed Tox21 fibers and zero median certifiable fraction in main Matbench and JARVIS audits. In held-out replays, certify-then-acquire reduced false decisions from 1.19% to 0.027% in Tox21 and from 20.3% to 0.128% in JARVIS, while changing model choice and identifying deployment-relevant probes. Deployment-ready benchmarks should report evidence, supported actions, ambiguity and completion cost rather than scores alone.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 8 of its 28 references that are in the map. First posted on arXiv 25 May 2026.
Builds on
8 of 28 references.
References · 8
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2020In Search of Lost Domain Generalizationcited by 1,054
- 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2016Distribution-Free Predictive Inference For Regressioncited by 668
- 2019Conformal Prediction Under Covariate Shiftcited by 519
Built on by
Citing papers per year: 2026: 1.
1 paper.