# A Survey of Online Experiment Design with the Stochastic Multi-Armed Bandit

arXiv: https://arxiv.org/abs/1510.00757 · HTML: https://mlsanity.com/paper/1510.00757/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2015.

## Built on by (most cited first, 18 of 18)

- [Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation](https://mlsanity.com/paper/2403.13578/) (2024)
- [Online Experimental Design With Estimation-Regret Trade-off Under Network Interference](https://mlsanity.com/paper/2412.03727/) (2024)
- [Evaluating Online Bandit Exploration In Large-Scale Recommender System](https://mlsanity.com/paper/2304.02572/) (2023)
- [Kolmogorov-Smirnov Test-Based Actively-Adaptive Thompson Sampling for Non-Stationary Bandits](https://mlsanity.com/paper/2105.14586/) (2021)
- [Contextual Active Model Selection](https://mlsanity.com/paper/2207.06030/) (2022)
- [Debiasing Samples from Online Learning Using Bootstrap](https://mlsanity.com/paper/2108.00236/) (2021)
- [Adaptive Behavioral AI: Reinforcement Learning to Enhance Pharmacy Services](https://mlsanity.com/paper/2408.07647/) (2024)
- [Existence conditions for hidden feedback loops in online recommender systems](https://mlsanity.com/paper/2109.05278/) (2021)
- [Multi-Armed Bandits in Brain-Computer Interfaces](https://mlsanity.com/paper/2205.09584/) (2022)
- [GROS: A General Robust Aggregation Strategy](https://mlsanity.com/paper/2402.15442/) (2024)
- [The Digital Transformation in Health: How AI Can Improve the Performance of Health Systems](https://mlsanity.com/paper/2409.16098/) (2024)
- [Follow-the-Perturbed-Leader for Decoupled Bandits: Best-of-Both-Worlds and Practicality](https://mlsanity.com/paper/2510.12152/) (2025)
- [TSEC: a framework for online experimentation under experimental constraints](https://mlsanity.com/paper/2101.06592/) (2021)
- [Adaptive Interventions for Global Health: A Case Study of Malaria](https://mlsanity.com/paper/2303.02075/) (2023)
- [Optimizing HIV Patient Engagement with Reinforcement Learning in Resource-Limited Settings](https://mlsanity.com/paper/2408.07629/) (2024)
- [Adaptive User Journeys in Pharma E-Commerce with Reinforcement Learning: Insights from SwipeRx](https://mlsanity.com/paper/2408.08024/) (2024)
- [A Green Multi-Attribute Client Selection for Over-The-Air Federated Learning: A Grey-Wolf-Optimizer Approach](https://mlsanity.com/paper/2409.11442/) (2024)
- [Information maximization for a broad variety of multi-armed bandit games](https://mlsanity.com/paper/2503.15962/) (2025)

## Abstract (arXiv)

Adaptive and sequential experiment design is a well-studied area in numerous domains. We survey and synthesize the work of the online statistical learning paradigm referred to as multi-armed bandits integrating the existing research as a resource for a certain class of online experiments. We first explore the traditional stochastic model of a multi-armed bandit, then explore a taxonomic scheme of complications to that model, for each complication relating it to a specific requirement or consideration of the experiment design context. Finally, at the end of the paper, we present a table of known upper-bounds of regret for all studied algorithms providing both perspectives for future theoretical work and a decision-making tool for practitioners looking for theoretical guarantees.