# Predictive Modeling through Hyper-Bayesian Optimization

arXiv: https://arxiv.org/abs/2308.00285 · HTML: https://mlsanity.com/paper/2308.00285/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 6 of 6)

- [Structure Discovery in Nonparametric Regression through Compositional Kernel Search](https://mlsanity.com/paper/1302.4922/) (2013)
- [Batched High-dimensional Bayesian Optimization via Structural Kernel Learning](https://mlsanity.com/paper/1703.01973/) (2017)
- [Accelerating Experimental Design by Incorporating Experimenter Hunches](https://mlsanity.com/paper/1907.09065/) (2019)
- [Monotone Function Estimation for Computer Experiments](https://mlsanity.com/paper/1309.3802/) (2013)
- [Adaptive Rate of Convergence of Thompson Sampling for Gaussian Process Optimization](https://mlsanity.com/paper/1705.06808/) (2017)
- [Bayesian Optimization Using Monotonicity Information and Its Application in Machine Learning Hyperparameter](https://mlsanity.com/paper/1802.03532/) (2018)

## Abstract (arXiv)

Model selection is an integral problem of model based optimization techniques such as Bayesian optimization (BO). Current approaches often treat model selection as an estimation problem, to be periodically updated with observations coming from the optimization iterations. In this paper, we propose an alternative way to achieve both efficiently. Specifically, we propose a novel way of integrating model selection and BO for the single goal of reaching the function optima faster. The algorithm moves back and forth between BO in the model space and BO in the function space, where the goodness of the recommended model is captured by a score function and fed back, capturing how well the model helped convergence in the function space. The score function is derived in such a way that it neutralizes the effect of the moving nature of the BO in the function space, thus keeping the model selection problem stationary. This back and forth leads to quick convergence for both model selection and BO in the function space. In addition to improved sample efficiency, the framework outputs information about the black-box function. Convergence is proved, and experimental results show significant improvement compared to standard BO.