# A Bayesian Optimization Framework for Finding Local Optima in Expensive Multi-Modal Functions

arXiv: https://arxiv.org/abs/2210.06635 · HTML: https://mlsanity.com/paper/2210.06635/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 67 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [Bayesian optimization for materials design](https://mlsanity.com/paper/1506.01349/) (2015)
- [A Stochastic Quasi-Newton Method for Large-Scale Optimization](https://mlsanity.com/paper/1401.7020/) (2014)
- [Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables](https://mlsanity.com/paper/1910.01688/) (2019)
- [Bayesian Optimization with Gradients](https://mlsanity.com/paper/1703.04389/) (2017)
- [Predictive Entropy Search for Bayesian Optimization with Unknown Constraints](https://mlsanity.com/paper/1502.05312/) (2015)
- [Efficient Nonmyopic Bayesian Optimization via One-Shot Multi-Step Trees](https://mlsanity.com/paper/2006.15779/) (2020)
- [GLASSES: Relieving The Myopia Of Bayesian Optimisation](https://mlsanity.com/paper/1510.06299/) (2015)
- [Local policy search with Bayesian optimization](https://mlsanity.com/paper/2106.11899/) (2021)
- [BINOCULARS for Efficient, Nonmyopic Sequential Experimental Design](https://mlsanity.com/paper/1909.04568/) (2019)
- [Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization](https://mlsanity.com/paper/1703.01250/) (2017)
- [Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian](https://mlsanity.com/paper/1605.09466/) (2016)
- [Network-Aware Optimization of Distributed Learning for Fog Computing](https://mlsanity.com/paper/2004.08488/) (2020)
- [Constrained Bayesian Optimization with Max-Value Entropy Search](https://mlsanity.com/paper/1910.07003/) (2019)
- [Practical Bayesian Optimization for Model Fitting with Bayesian Adaptive Direct Search](https://mlsanity.com/paper/1705.04405/) (2017)
- [Hierarchical ResNeXt Models for Breast Cancer Histology Image Classification](https://mlsanity.com/paper/1810.09025/) (2018)
- [Two-step Lookahead Bayesian Optimization with Inequality Constraints](https://mlsanity.com/paper/2112.02833/) (2021)

## Built on by (most cited first, 12 of 12)

- [AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning](https://mlsanity.com/paper/2406.18060/) (2024)
- [Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction](https://mlsanity.com/paper/2310.08670/) (2023)
- [MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization](https://mlsanity.com/paper/2302.10418/) (2023)
- [Collaborative AI Teaming in Unknown Environments via Active Goal Deduction](https://mlsanity.com/paper/2403.15341/) (2024)
- [Cooperative Backdoor Attack in Decentralized Reinforcement Learning with Theoretical Guarantee](https://mlsanity.com/paper/2405.15245/) (2024)
- [Bayesian Optimization through Gaussian Cox Process Models for Spatio-temporal Data](https://mlsanity.com/paper/2401.14544/) (2024)
- [Exploiting Partial Common Information Microstructure for Multi-Modal Brain Tumor Segmentation](https://mlsanity.com/paper/2302.02521/) (2023)
- [ReMIX: Regret Minimization for Monotonic Value Function Factorization in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2302.05593/) (2023)
- [RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space](https://mlsanity.com/paper/2410.16517/) (2024)
- [Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning](https://mlsanity.com/paper/2308.14897/) (2023)
- [A Balanced Approach of Rapid Genetic Exploration and Surrogate Exploitation for Hyperparameter Optimization](https://mlsanity.com/paper/2504.07359/) (2025)
- [Global Optimization on Graph-Structured Data via Gaussian Processes with Spectral Representations](https://mlsanity.com/paper/2511.07734/) (2025)

## Abstract (arXiv)

Bayesian optimization (BO) is a popular global optimization scheme for sample-efficient optimization in domains with expensive function evaluations. The existing BO techniques are capable of finding a single global optimum solution. However, finding a set of global and local optimum solutions is crucial in a wide range of real-world problems, as implementing some of the optimal solutions might not be feasible due to various practical restrictions (e.g., resource limitation, physical constraints, etc.). In such domains, if multiple solutions are known, the implementation can be quickly switched to another solution, and the best possible system performance can still be obtained. This paper develops a multimodal BO framework to effectively find a set of local/global solutions for expensive-to-evaluate multimodal objective functions. We consider the standard BO setting with Gaussian process regression representing the objective function. We analytically derive the joint distribution of the objective function and its first-order derivatives. This joint distribution is used in the body of the BO acquisition functions to search for local optima during the optimization process. We introduce variants of the well-known BO acquisition functions to the multimodal setting and demonstrate the performance of the proposed framework in locating a set of local optimum solutions using multiple optimization problems.