# Conservative Objective Models for Effective Offline Model-Based Optimization

arXiv: https://arxiv.org/abs/2107.06882 · HTML: https://mlsanity.com/paper/2107.06882/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [Conditioning by adaptive sampling for robust design](https://mlsanity.com/paper/1901.10060/) (2019)
- [Model Inversion Networks for Model-Based Optimization](https://mlsanity.com/paper/1912.13464/) (2019)
- [Chip Placement with Deep Reinforcement Learning](https://mlsanity.com/paper/2004.10746/) (2020)
- [Safe Controller Optimization for Quadrotors with Gaussian Processes](https://mlsanity.com/paper/1509.01066/) (2015)
- [The reparameterization trick for acquisition functions](https://mlsanity.com/paper/1712.00424/) (2017)
- [Autofocused oracles for model-based design](https://mlsanity.com/paper/2006.08052/) (2020)
- [Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation](https://mlsanity.com/paper/2102.07970/) (2021)
- [Population-Based Black-Box Optimization for Biological Sequence Design](https://mlsanity.com/paper/2006.03227/) (2020)
- [Data-efficient Learning of Morphology and Controller for a Microrobot](https://mlsanity.com/paper/1905.01334/) (2019)

## Built on by (most cited first, 30 of 88)

- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022)
- [Biological Sequence Design with GFlowNets](https://mlsanity.com/paper/2203.04115/) (2022)
- [Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization](https://mlsanity.com/paper/2202.08450/) (2022)
- [Diffusion Models for Black-Box Optimization](https://mlsanity.com/paper/2306.07180/) (2023)
- [Generative Pretraining for Black-Box Optimization](https://mlsanity.com/paper/2206.10786/) (2022)
- [Diffusion Models as Constrained Samplers for Optimization with Unknown Constraints](https://mlsanity.com/paper/2402.18012/) (2024)
- [Unsupervised Learning for Combinatorial Optimization with Principled Objective Relaxation](https://mlsanity.com/paper/2207.05984/) (2022)
- [Data-Driven Offline Decision-Making via Invariant Representation Learning](https://mlsanity.com/paper/2211.11349/) (2022)
- [Data-Driven Offline Optimization For Architecting Hardware Accelerators](https://mlsanity.com/paper/2110.11346/) (2021)
- [Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models](https://mlsanity.com/paper/2405.19673/) (2024)
- [Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences](https://mlsanity.com/paper/2306.03111/) (2023)
- [ExPT: Synthetic Pretraining for Few-Shot Experimental Design](https://mlsanity.com/paper/2310.19961/) (2023)
- [Improving Protein Optimization with Smoothed Fitness Landscapes](https://mlsanity.com/paper/2307.00494/) (2023)
- [Offline Model-Based Optimization via Policy-Guided Gradient Search](https://mlsanity.com/paper/2405.05349/) (2024)
- [Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC](https://mlsanity.com/paper/2212.09925/) (2022)
- [Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching](https://mlsanity.com/paper/2306.14079/) (2023)
- [ParetoFlow: Guided Flows in Multi-Objective Optimization](https://mlsanity.com/paper/2412.03718/) (2024)
- [Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits](https://mlsanity.com/paper/2306.07923/) (2023)
- [Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimization](https://mlsanity.com/paper/2407.01624/) (2024)
- [Offline Model-Based Optimization: Comprehensive Review](https://mlsanity.com/paper/2503.17286/) (2025)
- [Offline Multi-Objective Optimization](https://mlsanity.com/paper/2406.03722/) (2024)
- [Improved Off-policy Reinforcement Learning in Biological Sequence Design](https://mlsanity.com/paper/2410.04461/) (2024)
- [Offline Model-Based Optimization by Learning to Rank](https://mlsanity.com/paper/2410.11502/) (2024)
- [Preference-Guided Diffusion for Multi-Objective Offline Optimization](https://mlsanity.com/paper/2503.17299/) (2025)
- [Functional Graphical Models: Structure Enables Offline Data-Driven Optimization](https://mlsanity.com/paper/2401.05442/) (2024)
- [Monte Carlo Tree Search based Hybrid Optimization of Variational Quantum Circuits](https://mlsanity.com/paper/2203.16707/) (2022)
- [Generative Adversarial Model-Based Optimization via Source Critic Regularization](https://mlsanity.com/paper/2402.06532/) (2024)
- [Design Editing for Offline Model-based Optimization](https://mlsanity.com/paper/2405.13964/) (2024)
- [Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimization](https://mlsanity.com/paper/2306.14479/) (2023)

## Abstract (arXiv)

Computational design problems arise in a number of settings, from synthetic biology to computer architectures. In this paper, we aim to solve data-driven model-based optimization (MBO) problems, where the goal is to find a design input that maximizes an unknown objective function provided access to only a static dataset of prior experiments. Such data-driven optimization procedures are the only practical methods in many real-world domains where active data collection is expensive (e.g., when optimizing over proteins) or dangerous (e.g., when optimizing over aircraft designs). Typical methods for MBO that optimize the design against a learned model suffer from distributional shift: it is easy to find a design that "fools" the model into predicting a high value. To overcome this, we propose conservative objective models (COMs), a method that learns a model of the objective function that lower bounds the actual value of the ground-truth objective on out-of-distribution inputs, and uses it for optimization. Structurally, COMs resemble adversarial training methods used to overcome adversarial examples. COMs are simple to implement and outperform a number of existing methods on a wide range of MBO problems, including optimizing protein sequences, robot morphologies, neural network weights, and superconducting materials.