# Optimization of Epsilon-Greedy Exploration

arXiv: https://arxiv.org/abs/2506.03324 · HTML: https://mlsanity.com/paper/2506.03324/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 17 of 17)

- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Modern Bayesian Experimental Design](https://mlsanity.com/paper/2302.14545/) (2023)
- [Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design](https://mlsanity.com/paper/2103.02438/) (2021)
- [Off-policy evaluation for slate recommendation](https://mlsanity.com/paper/1605.04812/) (2016)
- [Batched Multi-armed Bandits Problem](https://mlsanity.com/paper/1904.01763/) (2019)
- [High-Dimensional Sparse Linear Bandits](https://mlsanity.com/paper/2011.04020/) (2020)
- [Regret Bounds for Batched Bandits](https://mlsanity.com/paper/1910.04959/) (2019)
- [On Explore-Then-Commit Strategies](https://mlsanity.com/paper/1605.08988/) (2016)
- [Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach](https://mlsanity.com/paper/1311.6355/) (2013)
- [Parallelizing Thompson Sampling](https://mlsanity.com/paper/2106.01420/) (2021)
- [Batched Thompson Sampling](https://mlsanity.com/paper/2110.00202/) (2021)
- [Local Policy Improvement for Recommender Systems](https://mlsanity.com/paper/2212.11431/) (2022)
- [Practical Bandits: An Industry Perspective](https://mlsanity.com/paper/2302.01223/) (2023)
- [Epsilon-Greedy Thompson Sampling to Bayesian Optimization](https://mlsanity.com/paper/2403.00540/) (2024)
- [Bandits Warm-up Cold Recommender Systems](https://mlsanity.com/paper/1407.2806/) (2014)
- [Adaptive Experimentation at Scale: A Computational Framework for Flexible Batches](https://mlsanity.com/paper/2303.11582/) (2023)
- [Optimization-Driven Adaptive Experimentation](https://mlsanity.com/paper/2408.04570/) (2024)

## Built on by (most cited first, 1 of 1)

- [Follow-the-Perturbed-Leader for Decoupled Bandits: Best-of-Both-Worlds and Practicality](https://mlsanity.com/paper/2510.12152/) (2025)

## Abstract (arXiv)

Modern recommendation systems rely on exploration to learn user preferences for new items, typically implementing uniform exploration policies (e.g., epsilon-greedy) due to their simplicity and compatibility with machine learning (ML) personalization models. Within these systems, a crucial consideration is the rate of exploration - what fraction of user traffic should receive random item recommendations and how this should evolve over time. While various heuristics exist for navigating the resulting exploration-exploitation tradeoff, selecting optimal exploration rates is complicated by practical constraints including batched updates, time-varying user traffic, short time horizons, and minimum exploration requirements. In this work, we propose a principled framework for determining the exploration schedule based on directly minimizing Bayesian regret through stochastic gradient descent (SGD), allowing for dynamic exploration rate adjustment via Model-Predictive Control (MPC). Through extensive experiments with recommendation datasets, we demonstrate that variations in the batch size across periods significantly influence the optimal exploration strategy. Our optimization methods automatically calibrate exploration to the specific problem setting, consistently matching or outperforming the best heuristic for each setting.