# Information-theoretic Bayesian Optimization: Survey and Tutorial

arXiv: https://arxiv.org/abs/2502.06789 · HTML: https://mlsanity.com/paper/2502.06789/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 27 of 27)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Max-value Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/1703.01968/) (2017)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [Freeze-Thaw Bayesian Optimization](https://mlsanity.com/paper/1406.3896/) (2014)
- [Predictive Entropy Search for Multi-objective Bayesian Optimization](https://mlsanity.com/paper/1511.05467/) (2015)
- [Max-value Entropy Search for Multi-Objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/2009.01721/) (2020)
- [Multi-objective Bayesian Optimization using Pareto-frontier Entropy](https://mlsanity.com/paper/1906.00127/) (2019)
- [Joint Entropy Search for Maximally-Informed Bayesian Optimization](https://mlsanity.com/paper/2206.04771/) (2022)
- [Predictive Entropy Search for Bayesian Optimization with Unknown Constraints](https://mlsanity.com/paper/1502.05312/) (2015)
- [Joint Entropy Search for Multi-objective Bayesian Optimization](https://mlsanity.com/paper/2210.02905/) (2022)
- [Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions](https://mlsanity.com/paper/1511.07130/) (2015)
- [Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/1609.01051/) (2016)
- [Noisy-Input Entropy Search for Efficient Robust Bayesian Optimization](https://mlsanity.com/paper/2002.02820/) (2020)
- [Multi-Fidelity Multi-Objective Bayesian Optimization: An Output Space Entropy Search Approach](https://mlsanity.com/paper/2011.01542/) (2020)
- [MUMBO: MUlti-task Max-value Bayesian Optimization](https://mlsanity.com/paper/2006.12093/) (2020)
- [Multi-fidelity Bayesian Optimization with Max-value Entropy Search and its parallelization](https://mlsanity.com/paper/1901.08275/) (2019)
- [Constrained Bayesian Optimization with Max-Value Entropy Search](https://mlsanity.com/paper/1910.07003/) (2019)
- [Fast Information-theoretic Bayesian Optimisation](https://mlsanity.com/paper/1711.00673/) (2017)
- [Improved Max-value Entropy Search for Multi-objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/2011.01150/) (2020)
- [Output Space Entropy Search Framework for Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2110.06980/) (2021)
- [An Entropy Search Portfolio for Bayesian Optimization](https://mlsanity.com/paper/1406.4625/) (2014)
- [Generalizing Bayesian Optimization with Decision-theoretic Entropies](https://mlsanity.com/paper/2210.01383/) (2022)
- [Trusted-Maximizers Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/2107.14465/) (2021)
- [Gaussian Max-Value Entropy Search for Multi-Agent Bayesian Optimization](https://mlsanity.com/paper/2303.05694/) (2023)
- [Rectified Max-Value Entropy Search for Bayesian Optimization](https://mlsanity.com/paper/2202.13597/) (2022)
- [Sequential Bayesian optimal experimental design for structural reliability analysis](https://mlsanity.com/paper/2007.00402/) (2020)
- [Alpha Entropy Search for New Information-based Bayesian Optimization](https://mlsanity.com/paper/2411.16586/) (2024)

## Built on by (most cited first, 2 of 2)

- [Reasoning by Commented Code for Table Question Answering](https://mlsanity.com/paper/2602.00543/) (2026)
- [Converge Then Diversify: Decoupling Convergence and Diversity in Multi-Objective Bayesian Optimisation](https://mlsanity.com/paper/2609.13396/) (2026)

## Abstract (arXiv)

Several scenarios require the optimization of non-convex black-box functions, that are noisy expensive to evaluate functions with unknown analytical expression, whose gradients are hence not accessible. For example, the hyper-parameter tuning problem of machine learning models. Bayesian optimization is a class of methods with state-of-the-art performance delivering a solution to this problem in real scenarios. It uses an iterative process that employs a probabilistic surrogate model, typically a Gaussian process, of the objective function to be optimized computing a posterior predictive distribution of the black-box function. Based on the information given by this posterior predictive distribution, Bayesian optimization includes the computation of an acquisition function that represents, for every input space point, the utility of evaluating that point in the next iteraiton if the objective of the process is to retrieve a global extremum. This paper is a survey of the information theoretical acquisition functions, whose performance typically outperforms the rest of acquisition functions. The main concepts of the field of information theory are also described in detail to make the reader aware of why information theory acquisition functions deliver great results in Bayesian optimization and how can we approximate them when they are intractable. We also cover how information theory acquisition functions can be adapted to complex optimization scenarios such as the multi-objective, constrained, non-myopic, multi-fidelity, parallel and asynchronous settings and provide further lines of research.