# Reliable algorithm selection for machine learning-guided design

arXiv: https://arxiv.org/abs/2503.20767 · HTML: https://mlsanity.com/paper/2503.20767/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 57 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Builds on (most cited first, 22 of 22)

- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [Estimating means of bounded random variables by betting](https://mlsanity.com/paper/2010.09686/) (2020)
- [Protein Design with Guided Discrete Diffusion](https://mlsanity.com/paper/2305.20009/) (2023)
- [Conditioning by adaptive sampling for robust design](https://mlsanity.com/paper/1901.10060/) (2019)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Selection by Prediction with Conformal p-values](https://mlsanity.com/paper/2210.01408/) (2022)
- [AdaLead: A simple and robust adaptive greedy search algorithm for sequence design](https://mlsanity.com/paper/2010.02141/) (2020)
- [Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design](https://mlsanity.com/paper/2202.03613/) (2022)
- [No-Regret Bayesian Optimization with Unknown Hyperparameters](https://mlsanity.com/paper/1901.03357/) (2019)
- [Design by adaptive sampling](https://mlsanity.com/paper/1810.03714/) (2018)
- [Model-free selective inference under covariate shift via weighted conformal p-values](https://mlsanity.com/paper/2307.09291/) (2023)
- [Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regression](https://mlsanity.com/paper/2305.00869/) (2023)
- [Improving Screening Processes via Calibrated Subset Selection](https://mlsanity.com/paper/2202.01147/) (2022)
- [Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimization](https://mlsanity.com/paper/2411.17983/) (2024)
- [Forecasting labels under distribution-shift for machine-guided sequence design](https://mlsanity.com/paper/2211.10422/) (2022)
- [Beyond the training set: an intuitive method for detecting distribution shift in model-based optimization](https://mlsanity.com/paper/2311.05363/) (2023)
- [Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient](https://mlsanity.com/paper/2405.18075/) (2024)
- [Is novelty predictable?](https://mlsanity.com/paper/2306.00872/) (2023)

## Built on by (most cited first, 2 of 2)

- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [ConfHit: Conformal Generative Design with Oracle Free Guarantees](https://mlsanity.com/paper/2603.07371/) (2026)

## Abstract (arXiv)

Algorithms for machine learning-guided design, or design algorithms, use machine learning-based predictions to propose novel objects with desired property values. Given a new design task -- for example, to design novel proteins with high binding affinity to a therapeutic target -- one must choose a design algorithm and specify any hyperparameters and predictive and/or generative models involved. How can these decisions be made such that the resulting designs are successful? This paper proposes a method for design algorithm selection, which aims to select design algorithms that will produce a distribution of design labels satisfying a user-specified success criterion -- for example, that at least ten percent of designs' labels exceed a threshold. It does so by combining designs' predicted property values with held-out labeled data to reliably forecast characteristics of the label distributions produced by different design algorithms, building upon techniques from prediction-powered inference. The method is guaranteed with high probability to return design algorithms that yield successful label distributions (or the null set if none exist), if the density ratios between the design and labeled data distributions are known. We demonstrate the method's effectiveness in simulated protein and RNA design tasks, in settings with either known or estimated density ratios.