# Deep Contextual Multi-armed Bandits

arXiv: https://arxiv.org/abs/1807.09809 · HTML: https://mlsanity.com/paper/1807.09809/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2018.

## Built on by (most cited first, 16 of 16)

- [Pervasive Machine Learning for Smart Radio Environments Enabled by Reconfigurable Intelligent Surfaces](https://mlsanity.com/paper/2205.03793/) (2022)
- [Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook](https://mlsanity.com/paper/2210.13623/) (2022)
- [Adapting to Non-Stationary Environments: Multi-Armed Bandit Enhanced Retrieval-Augmented Generation on Knowledge Graphs](https://mlsanity.com/paper/2412.07618/) (2024)
- [X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback](https://mlsanity.com/paper/2203.02072/) (2022)
- [Insurance pricing on price comparison websites via reinforcement learning](https://mlsanity.com/paper/2308.06935/) (2023)
- [Two-Stage Neural Contextual Bandits for Personalised News Recommendation](https://mlsanity.com/paper/2206.14648/) (2022)
- [An Efficient Algorithm for Deep Stochastic Contextual Bandits](https://mlsanity.com/paper/2104.05613/) (2021)
- [Empirical analysis of representation learning and exploration in neural kernel bandits](https://mlsanity.com/paper/2111.03543/) (2021)
- [Top-K Ranking Deep Contextual Bandits for Information Selection Systems](https://mlsanity.com/paper/2201.13287/) (2022)
- [Sequential Decision Making on Unmatched Data using Bayesian Kernel Embeddings](https://mlsanity.com/paper/2210.13692/) (2022)
- [A Reinforcement Learning Framework for Online Speaker Diarization](https://mlsanity.com/paper/2302.10924/) (2023)
- [Adaptive Endpointing with Deep Contextual Multi-armed Bandits](https://mlsanity.com/paper/2303.13407/) (2023)
- [Accelerating Matrix Diagonalization through Decision Transformers with Epsilon-Greedy Optimization](https://mlsanity.com/paper/2406.16191/) (2024)
- [Consistency of Selection Strategies for Fraud Detection](https://mlsanity.com/paper/2509.18739/) (2025)
- [Clutch Control: An Attention-based Combinatorial Bandit for Efficient Mutation in JavaScript Engine Fuzzing](https://mlsanity.com/paper/2510.12732/) (2025)
- [Piece of CAKE: Adaptive Execution Engines via Microsecond-Scale Learning](https://mlsanity.com/paper/2602.04181/) (2026)

## Abstract (arXiv)

Contextual multi-armed bandit problems arise frequently in important industrial applications. Existing solutions model the context either linearly, which enables uncertainty driven (principled) exploration, or non-linearly, by using epsilon-greedy exploration policies. Here we present a deep learning framework for contextual multi-armed bandits that is both non-linear and enables principled exploration at the same time. We tackle the exploration vs. exploitation trade-off through Thompson sampling by exploiting the connection between inference time dropout and sampling from the posterior over the weights of a Bayesian neural network. In order to adjust the level of exploration automatically as more data is made available to the model, the dropout rate is learned rather than considered a hyperparameter. We demonstrate that our approach substantially reduces regret on two tasks (the UCI Mushroom task and the Casino Parity task) when compared to 1) non-contextual bandits, 2) epsilon-greedy deep contextual bandits, and 3) fixed dropout rate deep contextual bandits. Our approach is currently being applied to marketing optimization problems at HubSpot.