# Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design

arXiv: https://arxiv.org/abs/2202.03613 · HTML: https://mlsanity.com/paper/2202.03613/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 70 references that are in the map.
- First posted on arXiv 8 Feb 2022.

## Builds on (most cited first, 28 of 28)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Strategic Classification](https://mlsanity.com/paper/1506.06980/) (2015)
- [Deep Reinforcement Learning for De-Novo Drug Design](https://mlsanity.com/paper/1711.10907/) (2017)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Learning by Transduction](https://mlsanity.com/paper/1301.7375/) (2013)
- [Conditioning by adaptive sampling for robust design](https://mlsanity.com/paper/1901.10060/) (2019)
- [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021)
- [Robust Validation: Confident Predictions Even When Distributions Shift](https://mlsanity.com/paper/2008.04267/) (2020)
- [AdaLead: A simple and robust adaptive greedy search algorithm for sequence design](https://mlsanity.com/paper/2010.02141/) (2020)
- [Autofocused oracles for model-based design](https://mlsanity.com/paper/2006.08052/) (2020)
- [Conditional molecular design with deep generative models](https://mlsanity.com/paper/1805.00108/) (2018)
- [Sample-Efficient Safety Assurances using Conformal Prediction](https://mlsanity.com/paper/2109.14082/) (2021)
- [Tracking the risk of a deployed model and detecting harmful distribution shifts](https://mlsanity.com/paper/2110.06177/) (2021)
- [Protein sequence design with deep generative models](https://mlsanity.com/paper/2104.04457/) (2021)
- [Generating and designing DNA with deep generative models](https://mlsanity.com/paper/1712.06148/) (2017)
- [iDECODe: In-distribution Equivariance for Conformal Out-of-distribution Detection](https://mlsanity.com/paper/2201.02331/) (2022)
- [PAC Confidence Predictions for Deep Neural Network Classifiers](https://mlsanity.com/paper/2011.00716/) (2020)
- [Testing for concept shift online](https://mlsanity.com/paper/2012.14246/) (2020)
- [A primer on model-guided exploration of fitness landscapes for biological sequence design](https://mlsanity.com/paper/2010.10614/) (2020)

## Built on by (most cited first, 30 of 54)

- [Conformal PID Control for Time Series Prediction](https://mlsanity.com/paper/2307.16895/) (2023)
- [Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets](https://mlsanity.com/paper/2207.02238/) (2022)
- [Conformal Prediction Regions for Time Series using Linear Complementarity Programming](https://mlsanity.com/paper/2304.01075/) (2023)
- [Non-Exchangeable Conformal Risk Control](https://mlsanity.com/paper/2310.01262/) (2023)
- [Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)](https://mlsanity.com/paper/2405.06627/) (2024)
- [Conformal Contextual Robust Optimization](https://mlsanity.com/paper/2310.10003/) (2023)
- [Conformal Prediction: A Data Perspective](https://mlsanity.com/paper/2410.06494/) (2024)
- [Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers](https://mlsanity.com/paper/2208.11111/) (2022)
- [Conformal Prediction for Causal Effects of Continuous Treatments](https://mlsanity.com/paper/2407.03094/) (2024)
- [Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Models](https://mlsanity.com/paper/2311.13628/) (2023)
- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [Conformal Drug Property Prediction with Density Estimation under Covariate Shift](https://mlsanity.com/paper/2310.12033/) (2023)
- [Certifiable 3D Object Pose Estimation: Foundations, Learning Models, and Self-Training](https://mlsanity.com/paper/2206.11215/) (2022)
- [Conformal Prediction under Levy-Prokhorov Distribution Shifts: Robustness to Local and Global Perturbations](https://mlsanity.com/paper/2502.14105/) (2025)
- [Bayesian Optimization of Antibodies Informed by a Generative Model of Evolving Sequences](https://mlsanity.com/paper/2412.07763/) (2024)
- [Predictive inference for time series: why is split conformal effective despite temporal dependence?](https://mlsanity.com/paper/2510.02471/) (2025)
- [What If We Had Used a Different App? Reliable Counterfactual KPI Analysis in Wireless Systems](https://mlsanity.com/paper/2410.00150/) (2024)
- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [Bridging Dimensions: Confident Reachability for High-Dimensional Controllers](https://mlsanity.com/paper/2311.04843/) (2023)
- [Conformal Predictive Systems Under Covariate Shift](https://mlsanity.com/paper/2404.15018/) (2024)
- [Distribution-Free Finite-Sample Guarantees and Split Conformal Prediction](https://mlsanity.com/paper/2210.14735/) (2022)
- [Beyond the training set: an intuitive method for detecting distribution shift in model-based optimization](https://mlsanity.com/paper/2311.05363/) (2023)
- [Multi-Modal Conformal Prediction Regions with Simple Structures by Optimizing Convex Shape Templates](https://mlsanity.com/paper/2312.07434/) (2023)
- [Conformal time series decomposition with component-wise exchangeability](https://mlsanity.com/paper/2406.16766/) (2024)
- [JANET: Joint Adaptive predictioN-region Estimation for Time-series](https://mlsanity.com/paper/2407.06390/) (2024)
- [Conformal Prediction Under Generalized Covariate Shift with Posterior Drift](https://mlsanity.com/paper/2502.17744/) (2025)
- [Non-exchangeable Conformal Prediction for Temporal Graph Neural Networks](https://mlsanity.com/paper/2507.02151/) (2025)
- [Conformal Prediction in The Loop: A Feedback-Based Uncertainty Model for Trajectory Optimization](https://mlsanity.com/paper/2510.16376/) (2025)
- [Knowing when to trust machine-learned interatomic potentials](https://mlsanity.com/paper/2605.00640/) (2026)
- [Design-based conformal prediction](https://mlsanity.com/paper/2303.01422/) (2023)

## Abstract (arXiv)

Many applications of machine learning methods involve an iterative protocol in which data are collected, a model is trained, and then outputs of that model are used to choose what data to consider next. For example, one data-driven approach for designing proteins is to train a regression model to predict the fitness of protein sequences, then use it to propose new sequences believed to exhibit greater fitness than observed in the training data. Since validating designed sequences in the wet lab is typically costly, it is important to quantify the uncertainty in the model's predictions. This is challenging because of a characteristic type of distribution shift between the training and test data in the design setting -- one in which the training and test data are statistically dependent, as the latter is chosen based on the former. Consequently, the model's error on the test data -- that is, the designed sequences -- has an unknown and possibly complex relationship with its error on the training data. We introduce a method to quantify predictive uncertainty in such settings. We do so by constructing confidence sets for predictions that account for the dependence between the training and test data. The confidence sets we construct have finite-sample guarantees that hold for any prediction algorithm, even when a trained model chooses the test-time input distribution. As a motivating use case, we demonstrate with several real data sets how our method quantifies uncertainty for the predicted fitness of designed proteins, and can therefore be used to select design algorithms that achieve acceptable trade-offs between high predicted fitness and low predictive uncertainty.