# Bayesian Optimization for Policy Search via Online-Offline Experimentation

arXiv: https://arxiv.org/abs/1904.01049 · HTML: https://mlsanity.com/paper/1904.01049/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2019.

## Built on by (most cited first, 20 of 20)

- [Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement](https://mlsanity.com/paper/2105.08195/) (2021)
- [Derivative-Free Reinforcement Learning: A Review](https://mlsanity.com/paper/2102.05710/) (2021)
- [No-Regret Algorithms for Private Gaussian Process Bandit Optimization](https://mlsanity.com/paper/2102.12467/) (2021)
- [Triangulation candidates for Bayesian optimization](https://mlsanity.com/paper/2112.07457/) (2021)
- [Looper: An end-to-end ML platform for product decisions](https://mlsanity.com/paper/2110.07554/) (2021)
- [Towards Safe Multi-Task Bayesian Optimization](https://mlsanity.com/paper/2312.07281/) (2023)
- [Amortized Auto-Tuning: Cost-Efficient Bayesian Transfer Optimization for Hyperparameter Recommendation](https://mlsanity.com/paper/2106.09179/) (2021)
- [On the Suboptimality of GP-UCB under Polynomial Effective Optimism](https://mlsanity.com/paper/2312.01386/) (2023)
- [Interpretable Personalized Experimentation](https://mlsanity.com/paper/2111.03267/) (2021)
- [Preferential Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2406.14699/) (2024)
- [Model Selection for Production System via Automated Online Experiments](https://mlsanity.com/paper/2105.13420/) (2021)
- [Optimal Initialization of Batch Bayesian Optimization](https://mlsanity.com/paper/2404.17997/) (2024)
- [Experimenting, Fast and Slow: Bayesian Optimization of Long-term Outcomes with Online Experiments](https://mlsanity.com/paper/2506.18744/) (2025)
- [Safe and Adaptive Decision-Making for Optimization of Safety-Critical Systems: The ARTEO Algorithm](https://mlsanity.com/paper/2211.05495/) (2022)
- [An Analysis of Safety Guarantees in Multi-Task Bayesian Optimization](https://mlsanity.com/paper/2503.08555/) (2025)
- [ExOSITO: Explainable Off-Policy Learning with Side Information for Intensive Care Unit Blood Test Orders](https://mlsanity.com/paper/2504.17277/) (2025)
- [Neural Forecasting of the Italian Sovereign Bond Market with Economic News](https://mlsanity.com/paper/2203.07071/) (2022)
- [Practical Policy Optimization with Personalized Experimentation](https://mlsanity.com/paper/2303.17648/) (2023)
- [A Constrained Multi-Fidelity Bayesian Optimization Method](https://mlsanity.com/paper/2510.10984/) (2025)
- [Offline-to-Online Creative Optimization with Generative Models and Adaptive Testing](https://mlsanity.com/paper/2607.23696/) (2026)

## Abstract (arXiv)

Online field experiments are the gold-standard way of evaluating changes to real-world interactive machine learning systems. Yet our ability to explore complex, multi-dimensional policy spaces - such as those found in recommendation and ranking problems - is often constrained by the limited number of experiments that can be run simultaneously. To alleviate these constraints, we augment online experiments with an offline simulator and apply multi-task Bayesian optimization to tune live machine learning systems. We describe practical issues that arise in these types of applications, including biases that arise from using a simulator and assumptions for the multi-task kernel. We measure empirical learning curves which show substantial gains from including data from biased offline experiments, and show how these learning curves are consistent with theoretical results for multi-task Gaussian process generalization. We find that improved kernel inference is a significant driver of multi-task generalization. Finally, we show several examples of Bayesian optimization efficiently tuning a live machine learning system by combining offline and online experiments.