# Surrogate modeling for Bayesian optimization beyond a single Gaussian process

arXiv: https://arxiv.org/abs/2205.14090 · HTML: https://mlsanity.com/paper/2205.14090/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 27 May 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [On Kernelized Multi-armed Bandits](https://mlsanity.com/paper/1704.00445/) (2017)
- [Learning to Optimize Via Posterior Sampling](https://mlsanity.com/paper/1301.2609/) (2013)
- [Max-value Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/1703.01968/) (2017)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [Efficiently Sampling Functions from Gaussian Process Posteriors](https://mlsanity.com/paper/2002.09309/) (2020)
- [Batched Large-scale Bayesian Optimization in High-dimensional Spaces](https://mlsanity.com/paper/1706.01445/) (2017)
- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendations](https://mlsanity.com/paper/1908.01425/) (2019)
- [Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space](https://mlsanity.com/paper/1706.01825/) (2017)
- [Parallel Bayesian Global Optimization of Expensive Functions](https://mlsanity.com/paper/1602.05149/) (2016)
- [Scalable Thompson Sampling using Sparse Gaussian Process Models](https://mlsanity.com/paper/2006.05356/) (2020)
- [Bayesian Optimization for Categorical and Category-Specific Continuous Inputs](https://mlsanity.com/paper/1911.12473/) (2019)
- [Scaling up the Automatic Statistician: Scalable Structure Discovery using Gaussian Processes](https://mlsanity.com/paper/1706.02524/) (2017)
- [Thompson Sampling with a Mixture Prior](https://mlsanity.com/paper/2106.05608/) (2021)
- [Incremental Ensemble Gaussian Processes](https://mlsanity.com/paper/2110.06777/) (2021)
- [An Entropy Search Portfolio for Bayesian Optimization](https://mlsanity.com/paper/1406.4625/) (2014)
- [Scalable Variational Bayesian Kernel Selection for Sparse Gaussian Process Regression](https://mlsanity.com/paper/1912.02641/) (2019)
- [Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes](https://mlsanity.com/paper/2112.00882/) (2021)

## Built on by (most cited first, 22 of 22)

- [Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs](https://mlsanity.com/paper/2509.17998/) (2025)
- [ActiveInitSplat: How Active Image Selection Helps Gaussian Splatting](https://mlsanity.com/paper/2503.06859/) (2025)
- [Dynamic Online Ensembles of Basis Expansions](https://mlsanity.com/paper/2405.01365/) (2024)
- [Online scalable Gaussian processes with conformal prediction for guaranteed coverage](https://mlsanity.com/paper/2410.05444/) (2024)
- [Low-Rank Adaptation Redux for Large Models](https://mlsanity.com/paper/2604.21905/) (2026)
- [Weighted Ensembles for Active Learning with Adaptivity](https://mlsanity.com/paper/2206.05009/) (2022)
- [Adaptive Prior Selection in Gaussian Process Bandits with Thompson Sampling](https://mlsanity.com/paper/2502.01226/) (2025)
- [Fine-tuning LLMs with variational Bayesian last layer for high-dimensional Bayesian optimization](https://mlsanity.com/paper/2510.01471/) (2025)
- [3D Reconstruction in Noisy Agricultural Environments: A Bayesian Optimization Perspective for View Planning](https://mlsanity.com/paper/2310.00145/) (2023)
- [Deploying AI for Signal Processing education: Selected challenges and intriguing opportunities](https://mlsanity.com/paper/2509.08950/) (2025)
- [BOSfM: A View Planning Framework for Optimal 3D Reconstruction of Agricultural Scenes](https://mlsanity.com/paper/2509.24126/) (2025)
- [Bayes-Split-Edge: Bayesian Optimization for Constrained Collaborative Inference in Wireless Edge Systems](https://mlsanity.com/paper/2510.23503/) (2025)
- [Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity](https://mlsanity.com/paper/2608.24721/) (2026)
- [MolLIBRA: Genetic Molecular Optimization with Multi-Fingerprint Surrogates and Text-Molecule Aligned Critic](https://mlsanity.com/paper/2602.07002/) (2026)
- [3D Scene Rendering with Multimodal Gaussian Splatting](https://mlsanity.com/paper/2602.17124/) (2026)
- [A Single Image and Multimodality Is All You Need for Novel View Synthesis](https://mlsanity.com/paper/2602.17909/) (2026)
- [ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization](https://mlsanity.com/paper/2605.06454/) (2026)
- [BASIL: Bayesian Application for Scientific Iteration and Learning](https://mlsanity.com/paper/2606.21092/) (2026)
- [LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms](https://mlsanity.com/paper/2607.08791/) (2026)
- [Bayesian optimization with kernel ensembles and disagreement-based acquisition for source localization and acoustic inversion](https://mlsanity.com/paper/2609.14262/) (2026)
- [Online Adaptive Kernel Mixing for Gaussian Process Decision Making](https://mlsanity.com/paper/2609.19891/) (2026)
- [AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models](https://mlsanity.com/paper/2609.36368/) (2026)

## Abstract (arXiv)

Bayesian optimization (BO) has well-documented merits for optimizing black-box functions with an expensive evaluation cost. Such functions emerge in applications as diverse as hyperparameter tuning, drug discovery, and robotics. BO hinges on a Bayesian surrogate model to sequentially select query points so as to balance exploration with exploitation of the search space. Most existing works rely on a single Gaussian process (GP) based surrogate model, where the kernel function form is typically preselected using domain knowledge. To bypass such a design process, this paper leverages an ensemble (E) of GPs to adaptively select the surrogate model fit on-the-fly, yielding a GP mixture posterior with enhanced expressiveness for the sought function. Acquisition of the next evaluation input using this EGP-based function posterior is then enabled by Thompson sampling (TS) that requires no additional design parameters. To endow function sampling with scalability, random feature-based kernel approximation is leveraged per GP model. The novel EGP-TS readily accommodates parallel operation. To further establish convergence of the proposed EGP-TS to the global optimum, analysis is conducted based on the notion of Bayesian regret for both sequential and parallel settings. Tests on synthetic functions and real-world applications showcase the merits of the proposed method.