# Bayesian Optimization by Kernel Regression and Density-based Exploration

arXiv: https://arxiv.org/abs/2502.06178 · HTML: https://mlsanity.com/paper/2502.06178/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 72 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 15 of 15)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [A Literature Survey of Benchmark Functions For Global Optimization Problems](https://mlsanity.com/paper/1308.4008/) (2013)
- [Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation](https://mlsanity.com/paper/1911.12809/) (2019)
- [BORE: Bayesian Optimization by Density-Ratio Estimation](https://mlsanity.com/paper/2102.09009/) (2021)
- [IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics](https://mlsanity.com/paper/2111.04077/) (2021)
- [A General Recipe for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2206.13035/) (2022)
- [Global optimization via inverse distance weighting and radial basis functions](https://mlsanity.com/paper/1906.06498/) (2019)
- [Bayesian Optimized Monte Carlo Planning](https://mlsanity.com/paper/2010.03597/) (2020)
- [Batch Bayesian optimisation via density-ratio estimation with guarantees](https://mlsanity.com/paper/2209.10715/) (2022)
- [Ada-BKB: Scalable Gaussian Process Optimization on Continuous Domains by Adaptive Discretization](https://mlsanity.com/paper/2106.08598/) (2021)
- [GLISp-r: A preference-based optimization algorithm with convergence guarantees](https://mlsanity.com/paper/2202.01125/) (2022)
- [Sequential adaptive design for emulating costly computer codes](https://mlsanity.com/paper/2206.12113/) (2022)

## Abstract (arXiv)

Bayesian optimization is highly effective for optimizing expensive-to-evaluate black-box functions, but it faces significant computational challenges due to the cubic per-iteration cost of Gaussian processes, which results in a total time complexity that is quartic with respect to the number of iterations. To address this limitation, we propose a novel algorithm, Bayesian optimization by kernel regression and density-based exploration (BOKE). BOKE uses kernel regression for efficient function approximation, kernel density for exploration, and integrates them into the confidence bound criteria to guide the optimization process, thus reducing computational costs to quadratic. Our theoretical analysis rigorously establishes the global convergence of BOKE under noisy evaluations. Through extensive numerical experiments on both synthetic and real-world optimization tasks, we demonstrate that BOKE not only performs competitively compared to Gaussian process-based methods and several other baseline methods but also exhibits superior computational efficiency. These results highlight BOKE's effectiveness in resource-constrained environments, providing a practical approach for optimization problems in engineering applications.