mlsanityData to 7 Oct 2026

When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design

arXiv 2608.01378 ↗First posted 2 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Design campaigns in chemistry, materials science, and machine learning share a bottleneck: determining how good a candidate truly is requires an expensive evaluation - an experiment, a first-principles simulation, or a full training run. Machine-learning surrogates that predict these outcomes are increasingly used not only to propose candidates but to grade them, and even to feed their own predictions back into the search as though they were measurements. Through mathematical analysis validated on three exhaustively ground-truthed design tasks, we establish when this practice is safe, what any certificate of safety must cost, and when the substitution provably pays. Predictive accuracy cannot anchor trust: near-perfect R^2 is compatible with worst-possible selections, and screening N candidates inflates the over-prediction at the selected candidate by a quantifiable "selection tax" with matching upper and lower bounds. Safety follows instead from an architectural rule - predictions may propose and train without restriction, but every certified conclusion must rest on true evaluations - which is sufficient with no assumptions on the surrogate, and necessary, since admitting predictions into certification with the standing of measurements opens a deterministic self-confirmation failure mode. We derive the minimal criterion under which a model may act as an oracle (rank preservation, not accuracy), show that trust must be purchased through selection-aware audits that are optimal in query complexity, and prove a dichotomy fixing when audited surrogates cut certified evaluation cost. Across 432 surrogate fits over six task-regime conditions, the audit statistic tracks deployed search performance at Spearman rank correlation 0.80-0.99, while the rank correlation of R^2 with deployed regret falls as low as 0.33; audited screening reduces certified oracle cost by a measured factor of 25.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 73 references that are in the map. First posted on arXiv 2 Aug 2026.

Builds on

42 of 73 references.

References · 42

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2017Selective Classification for Deep Neural Networkscited by 799
  3. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  4. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  5. 2017On Kernelized Multi-armed Banditscited by 348
  6. 2017Smart "Predict, then Optimize"cited by 332
  7. 2014On the Complexity of Best Arm Identification in Multi-Armed Bandit Modelscited by 280
  8. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  9. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 220
  10. 2013lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Banditscited by 148
  11. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  12. 2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 104
30 more
  1. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 96
  2. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 88
  3. 2021Conservative Objective Models for Effective Offline Model-Based Optimizationcited by 87
  4. 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
  5. 2018Mixture Martingales Revisited with Applications to Sequential Tests and Confidence Intervalscited by 76
  6. 2022Selection by Prediction with Conformal p-valuescited by 66
  7. 2022Conformal Prediction Under Feedback Covariate Shift for Biomolecular Designcited by 53
  8. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
  9. 2019The True Sample Complexity of Identifying Good Armscited by 37
  10. 2016Multi-fidelity Gaussian Process Bandit Optimisationcited by 31
  11. 2018Pure Exploration in Infinitely-Armed Bandit Models with Fixed-Confidencecited by 15
  12. 2021The Perils of Learning Before Optimizingcited by 15
  13. 2024Confidence on the Focal: Conformal Prediction with Selection-Conditional Coveragecited by 14
  14. 2024Offline Model-Based Optimization by Learning to Rankcited by 13
  15. 2025Multi-Armed Bandits With Machine Learning-Generated Surrogate Rewardscited by 13
  16. 2023Selective Sampling and Imitation Learning via Online Regressioncited by 12
  17. 2024Optimal Multi-Fidelity Best-Arm Identificationcited by 8
  18. 2026Best Arm Identification with LLM Judges and Limited Humancited by 7
  19. 2026When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewardscited by 5
  20. 2026Predicting LLM Safety Before Release by Simulating Deploymentcited by 5
  21. 2023Optimizer's Information Criterion: Dissecting and Correcting Bias in Data-Driven Optimizationcited by 4
  22. 2025Overtuning in Hyperparameter Optimizationcited by 4
  23. 2026Fundamental Limits of Black-Box Safety Evaluation: Information-Theoretic and Computational Barriers from Latent Context Conditioningcited by 3
  24. 2024Stabilizing black-box model selection with the inflated argmaxcited by 2
  25. 2026On the Learnability of Offline Model-Based Optimization: A Ranking Perspectivecited by 2
  26. 2026Flexible Inference for Winners with Conditional Validitycited by 2
  27. 2023Efficient estimation and correction of selection-induced bias with order statisticscited by 1
  28. 2026Conformal Candidate Certification for Offline Model-Based Optimizationcited by 1
  29. 2026Best-Arm Identification with Generative Proxycited by 1
  30. 2026Finite-Sample Coverage Audits for High-Recall Candidate Generation: Certification and Learning-Theoretic Designcited by 1

Built on by

Not cited yet.