mlsanityData to 14 Oct 2026

Deployment-complete benchmarking

arXiv 2605.25997 ↗First posted 25 May 2026cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

Benchmarks increasingly guide deployment, procurement and scientific screening, yet a score supports only the response it records, not necessarily the deployment action. We introduce deployment-complete benchmarking, which tests whether benchmark evidence determines a deployment action. A benchmark is complete for a claim exactly when the action is constant on each evidence fiber; mixed fibers expose missing deployment information, and completion curves quantify the evidence required to resolve ambiguity. In controlled response spaces, benchmark-channel conformal coverage of 94.98% transferred poorly to an unmeasured deployment channel (10.07%), whereas response-rank intervals achieved 94.91% coverage; even zero benchmark error certified only 45.4% of candidates at the largest residual size. Public audits revealed incompleteness, including 97.9% mixed Tox21 fibers and zero median certifiable fraction in main Matbench and JARVIS audits. In held-out replays, certify-then-acquire reduced false decisions from 1.19% to 0.027% in Tox21 and from 20.3% to 0.128% in JARVIS, while changing model choice and identifying deployment-relevant probes. Deployment-ready benchmarks should report evidence, supported actions, ambiguity and completion cost rather than scores alone.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 8 of its 28 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

8 of 28 references.

References · 8

  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  2. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  3. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  4. 2020In Search of Lost Domain Generalizationcited by 1,054
  5. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  6. 2017Selective Classification for Deep Neural Networkscited by 852
  7. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  8. 2019Conformal Prediction Under Covariate Shiftcited by 519

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.