# Epsilon-Greedy Thompson Sampling to Bayesian Optimization

arXiv: https://arxiv.org/abs/2403.00540 · HTML: https://mlsanity.com/paper/2403.00540/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 59 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Learning to Optimize Via Posterior Sampling](https://mlsanity.com/paper/1301.2609/) (2013)
- [Max-value Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/1703.01968/) (2017)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Efficiently Sampling Functions from Gaussian Process Posteriors](https://mlsanity.com/paper/2002.09309/) (2020)
- [Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables](https://mlsanity.com/paper/1910.01688/) (2019)
- [Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation](https://mlsanity.com/paper/1911.12809/) (2019)
- [Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming](https://mlsanity.com/paper/2004.11408/) (2020)
- [Multi-fidelity Bayesian Optimization: A Review](https://mlsanity.com/paper/2311.13050/) (2023)
- [Accelerating Approximate Thompson Sampling with Underdamped Langevin Monte Carlo](https://mlsanity.com/paper/2401.11665/) (2024)

## Built on by (most cited first, 9 of 9)

- [Exploring Exploration in Bayesian Optimization](https://mlsanity.com/paper/2502.08208/) (2025)
- [Automated design of nonreciprocal thermal emitters via Bayesian optimization](https://mlsanity.com/paper/2409.09192/) (2024)
- [Multi-fidelity Machine Learning for Uncertainty Quantification and Optimization](https://mlsanity.com/paper/2410.23482/) (2024)
- [Sampling from Gaussian Processes: A Tutorial and Applications in Global Sensitivity Analysis and Optimization](https://mlsanity.com/paper/2507.14746/) (2025)
- [Optimization of Epsilon-Greedy Exploration](https://mlsanity.com/paper/2506.03324/) (2025)
- [Bayesian Optimization under Uncertainty for Training a Scale Parameter in Stochastic Models](https://mlsanity.com/paper/2510.06439/) (2025)
- [TextBandit: Evaluating Probabilistic Reasoning in LLMs Through Language-Only Decision Tasks](https://mlsanity.com/paper/2510.13878/) (2025)
- [Digital Twins and Civil Engineering Phases: Reorienting Adoption Strategies](https://mlsanity.com/paper/2403.02426/) (2024)
- [Exploration strategies for Gaussian process bandit optimization with inference regret guarantees](https://mlsanity.com/paper/2610.16187/) (2026)

## Abstract (arXiv)

Bayesian optimization (BO) has become a powerful tool for solving simulation-based engineering optimization problems thanks to its ability to integrate physical and mathematical understandings, consider uncertainty, and address the exploitation-exploration dilemma. Thompson sampling (TS) is a preferred solution for BO to handle the exploitation-exploration trade-off. While it prioritizes exploration by generating and minimizing random sample paths from probabilistic models -- a fundamental ingredient of BO -- TS weakly manages exploitation by gathering information about the true objective function after it obtains new observations. In this work, we improve the exploitation of TS by incorporating the varepsilon-greedy policy, a well-established selection strategy in reinforcement learning. We first delineate two extremes of TS, namely the generic TS and the sample-average TS. The former promotes exploration, while the latter favors exploitation. We then adopt the varepsilon-greedy policy to randomly switch between these two extremes. Small and large values of varepsilon govern exploitation and exploration, respectively. By minimizing two benchmark functions and solving an inverse problem of a steel cantilever beam, we empirically show that varepsilon-greedy TS equipped with an appropriate varepsilon is more robust than its two extremes, matching or outperforming the better of the generic TS and the sample-average TS.