# Non-smooth Bayesian Optimization in Tuning Problems

arXiv: https://arxiv.org/abs/2109.07563 · HTML: https://mlsanity.com/paper/2109.07563/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 67 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [Multi-fidelity Bayesian Optimisation with Continuous Approximations](https://mlsanity.com/paper/1703.06240/) (2017)
- [Constrained Bayesian Optimization with Noisy Experiments](https://mlsanity.com/paper/1706.07094/) (2017)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [Input Warping for Bayesian Optimization of Non-stationary Functions](https://mlsanity.com/paper/1402.0929/) (2014)
- [Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes](https://mlsanity.com/paper/1706.03673/) (2017)
- [Local Gaussian process approximation for large computer experiments](https://mlsanity.com/paper/1303.0383/) (2013)
- [Bayesian Optimisation over Multiple Continuous and Categorical Inputs](https://mlsanity.com/paper/1906.08878/) (2019)
- [A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes](https://mlsanity.com/paper/1811.00755/) (2018)
- [Student-t Processes as Alternatives to Gaussian Processes](https://mlsanity.com/paper/1402.4306/) (2014)
- [Cost-aware Bayesian Optimization](https://mlsanity.com/paper/2003.10870/) (2020)
- [Convergence Guarantees for Gaussian Process Means With Misspecified Likelihoods and Smoothness](https://mlsanity.com/paper/2001.10818/) (2020)
- [Convex Clustering: Model, Theoretical Guarantee and Efficient Algorithm](https://mlsanity.com/paper/1810.02677/) (2018)
- [Bayesian Optimization for Iterative Learning](https://mlsanity.com/paper/1909.09593/) (2019)
- [Multitask and Transfer Learning for Autotuning Exascale Applications](https://mlsanity.com/paper/1908.05792/) (2019)
- [Scalable Gaussian Processes for Characterizing Multidimensional Change Surfaces](https://mlsanity.com/paper/1511.04408/) (2015)
- [Sequential Design for Optimal Stopping Problems](https://mlsanity.com/paper/1309.3832/) (2013)

## Built on by (most cited first, 8 of 8)

- [Sharded Bayesian Additive Regression Trees](https://mlsanity.com/paper/2306.00361/) (2023)
- [SANE: Strategic Autonomous Non-Smooth Exploration for Multiple Optima Discovery in Multi-modal and Non-differentiable Black-box Functions](https://mlsanity.com/paper/2409.12295/) (2024)
- [Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization](https://mlsanity.com/paper/2206.01409/) (2022)
- [Application Performance Modeling via Tensor Completion](https://mlsanity.com/paper/2210.10184/) (2022)
- [Nonparametric Multi-shape Modeling with Uncertainty Quantification](https://mlsanity.com/paper/2206.09127/) (2022)
- [Efficient and Robust Bayesian Selection of Hyperparameters in Dimension Reduction for Visualization](https://mlsanity.com/paper/2306.00357/) (2023)
- [Efficient Decision Trees for Tensor Regressions](https://mlsanity.com/paper/2408.01926/) (2024)
- [Gradient-based Sample Selection for Faster Bayesian Optimization](https://mlsanity.com/paper/2504.07742/) (2025)

## Abstract (arXiv)

Building surrogate models is one common approach when we attempt to learn unknown black-box functions. Bayesian optimization provides a framework which allows us to build surrogate models based on sequential samples drawn from the function and find the optimum. Tuning algorithmic parameters to optimize the performance of large, complicated "black-box" application codes is a specific important application, which aims at finding the optima of black-box functions. Within the Bayesian optimization framework, the Gaussian process model produces smooth or continuous sample paths. However, the black-box function in the tuning problem is often non-smooth. This difficult tuning problem is worsened by the fact that we usually have limited sequential samples from the black-box function. Motivated by these issues encountered in tuning, we propose a novel additive Gaussian process model called clustered Gaussian process (cGP), where the additive components are induced by clustering. In the examples we studied, the performance can be improved by as much as 90% among repetitive experiments. By using this surrogate model, we want to capture the non-smoothness of the black-box function. In addition to an algorithm for constructing this model, we also apply the model to several artificial and real applications to evaluate it.