# BORE: Bayesian Optimization by Density-Ratio Estimation

arXiv: https://arxiv.org/abs/2102.09009 · HTML: https://mlsanity.com/paper/2102.09009/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 95 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Max-value Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/1703.01968/) (2017)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [Hyperparameter Ensembles for Robustness and Uncertainty Quantification](https://mlsanity.com/paper/2006.13570/) (2020)
- [Multi-fidelity Bayesian Optimisation with Continuous Approximations](https://mlsanity.com/paper/1703.06240/) (2017)
- [Batched Large-scale Bayesian Optimization in High-dimensional Spaces](https://mlsanity.com/paper/1706.01445/) (2017)
- [Maximizing acquisition functions for Bayesian optimization](https://mlsanity.com/paper/1805.10196/) (2018)
- [Predictive Entropy Search for Multi-objective Bayesian Optimization](https://mlsanity.com/paper/1511.05467/) (2015)
- [Likelihood-free inference by ratio estimation](https://mlsanity.com/paper/1611.10242/) (2016)
- [Input Warping for Bayesian Optimization of Non-stationary Functions](https://mlsanity.com/paper/1402.0929/) (2014)
- [Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes](https://mlsanity.com/paper/1706.03673/) (2017)
- [Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization](https://mlsanity.com/paper/1905.04970/) (2019)
- [Optimization-Based Autonomous Racing of 1:43 Scale RC Cars](https://mlsanity.com/paper/1711.07300/) (2017)
- [Efficient Bayesian Experimental Design for Implicit Models](https://mlsanity.com/paper/1810.09912/) (2018)
- [Meta-Surrogate Benchmarking for Hyperparameter Optimization](https://mlsanity.com/paper/1905.12982/) (2019)
- [Learning How to Autonomously Race a Car: a Predictive Control Approach](https://mlsanity.com/paper/1901.08184/) (2019)
- [Computing the racing line using Bayesian optimization](https://mlsanity.com/paper/2002.04794/) (2020)

## Built on by (most cited first, 26 of 26)

- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [Automatic Termination for Hyperparameter Optimization](https://mlsanity.com/paper/2104.08166/) (2021)
- [A General Recipe for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2206.13035/) (2022)
- [Optimizing Hyperparameters with Conformal Quantile Regression](https://mlsanity.com/paper/2305.03623/) (2023)
- [KGTuner: Efficient Hyper-parameter Search for Knowledge Graph Learning](https://mlsanity.com/paper/2205.02460/) (2022)
- [Variational Search Distributions](https://mlsanity.com/paper/2409.06142/) (2024)
- [Bayesian Optimization For Multi-Objective Mixed-Variable Problems](https://mlsanity.com/paper/2201.12767/) (2022)
- [MBORE: Multi-objective Bayesian Optimisation by Density-Ratio Estimation](https://mlsanity.com/paper/2203.16912/) (2022)
- [Pseudo-Bayesian Optimization](https://mlsanity.com/paper/2310.09766/) (2023)
- [Efficient and Joint Hyperparameter and Architecture Search for Collaborative Filtering](https://mlsanity.com/paper/2307.11004/) (2023)
- [carps: A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks](https://mlsanity.com/paper/2506.06143/) (2025)
- [Uncertainty-Aware Multi-Objective Reinforcement Learning-Guided Diffusion Models for 3D De Novo Molecular Design](https://mlsanity.com/paper/2510.21153/) (2025)
- [Generative Bayesian Optimization: Generative Models as Acquisition Functions](https://mlsanity.com/paper/2510.25240/) (2025)
- [MALIBO: Meta-learning for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2307.03565/) (2023)
- [On the development of a practical Bayesian optimisation algorithm for expensive experiments and simulations with changing environmental conditions](https://mlsanity.com/paper/2402.03006/) (2024)
- [Obeying the Order: Introducing Ordered Transfer Hyperparameter Optimisation](https://mlsanity.com/paper/2306.16916/) (2023)
- [Regret Analysis of Guided Diffusion for Black-Box Optimization over Structured Inputs](https://mlsanity.com/paper/2605.10385/) (2026)
- [Kernel-based guarantees for nonlinear parametric models in Bayesian optimization](https://mlsanity.com/paper/2605.13160/) (2026)
- [Searching in the Forest for Local Bayesian Optimization](https://mlsanity.com/paper/2111.05834/) (2021)
- [Density Ratio Estimation-based Bayesian Optimization with Semi-Supervised Learning](https://mlsanity.com/paper/2305.15612/) (2023)
- [Deterministic Langevin Unconstrained Optimization with Normalizing Flows](https://mlsanity.com/paper/2310.00745/) (2023)
- [Poisson Process for Bayesian Optimization](https://mlsanity.com/paper/2402.02687/) (2024)
- [Bayesian Optimization by Kernel Regression and Density-based Exploration](https://mlsanity.com/paper/2502.06178/) (2025)
- [Informing Acquisition Functions via Foundation Models for Molecular Discovery](https://mlsanity.com/paper/2512.13935/) (2025)
- [An Open-Source Training Dataset for Foundation Models for Black-box Optimization](https://mlsanity.com/paper/2605.23417/) (2026)
- [Bandits in Prod: Hyperparameter Optimization at Inference Time](https://mlsanity.com/paper/2609.01335/) (2026)

## Abstract (arXiv)

Bayesian optimization (BO) is among the most effective and widely-used blackbox optimization methods. BO proposes solutions according to an explore-exploit trade-off criterion encoded in an acquisition function, many of which are computed from the posterior predictive of a probabilistic surrogate model. Prevalent among these is the expected improvement (EI) function. The need to ensure analytical tractability of the predictive often poses limitations that can hinder the efficiency and applicability of BO. In this paper, we cast the computation of EI as a binary classification problem, building on the link between class-probability estimation and density-ratio estimation, and the lesser-known link between density-ratios and EI. By circumventing the tractability constraints, this reformulation provides numerous advantages, not least in terms of expressiveness, versatility, and scalability.