# Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning

arXiv: https://arxiv.org/abs/2011.14266 · HTML: https://mlsanity.com/paper/2011.14266/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 29 of its 108 references that are in the map.
- First posted on arXiv 29 Nov 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 29 of 29)

- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Learning to Optimize Via Posterior Sampling](https://mlsanity.com/paper/1301.2609/) (2013)
- [An Information-Theoretic Analysis of Thompson Sampling](https://mlsanity.com/paper/1403.5341/) (2014)
- [Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)](https://mlsanity.com/paper/1503.01057/) (2015)
- [Noisy Natural Gradient as Variational Inference](https://mlsanity.com/paper/1712.02390/) (2017)
- [Ensemble Sampling](https://mlsanity.com/paper/1705.07347/) (2017)
- [Horizon: Facebook's Open Source Applied Reinforcement Learning Platform](https://mlsanity.com/paper/1811.00260/) (2018)
- [Bayesian Dark Knowledge](https://mlsanity.com/paper/1506.04416/) (2015)
- [Thompson Sampling for Complex Bandit Problems](https://mlsanity.com/paper/1311.0466/) (2013)
- [An Efficient Bandit Algorithm for Realtime Multivariate Optimization](https://mlsanity.com/paper/1810.09558/) (2018)
- [Thoughts on Massively Scalable Gaussian Processes](https://mlsanity.com/paper/1511.01870/) (2015)
- [Multi-scale exploration of convex functions and bandit convex optimization](https://mlsanity.com/paper/1507.06580/) (2015)
- [Thompson sampling with the online bootstrap](https://mlsanity.com/paper/1410.4009/) (2014)
- [Designing and Deploying Online Field Experiments](https://mlsanity.com/paper/1409.3174/) (2014)
- [Simple, Distributed, and Accelerated Probabilistic Programming](https://mlsanity.com/paper/1811.02091/) (2018)
- [On the Prior Sensitivity of Thompson Sampling](https://mlsanity.com/paper/1506.03378/) (2015)
- [Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT](https://mlsanity.com/paper/1907.11804/) (2019)
- [Real-world Video Adaptation with Reinforcement Learning](https://mlsanity.com/paper/2008.12858/) (2020)
- [Online Learning to Estimate Warfarin Dose with Contextual Linear Bandits](https://mlsanity.com/paper/1907.05496/) (2019)

## Built on by (most cited first, 1 of 1)

- [AExGym: Benchmarks and Environments for Adaptive Experimentation](https://mlsanity.com/paper/2408.04531/) (2024)

## Abstract (arXiv)

Thompson sampling (TS) has emerged as a robust technique for contextual bandit problems. However, TS requires posterior inference and optimization for action generation, prohibiting its use in many online platforms where latency and ease of deployment are of concern. We operationalize TS by proposing a novel imitation-learning-based algorithm that distills a TS policy into an explicit policy representation, allowing fast decision-making and easy deployment in mobile and server-based environments. Using batched data collected under the imitation policy, our algorithm iteratively performs offline updates to the TS policy, and learns a new explicit policy representation to imitate it. Empirically, our imitation policy achieves performance comparable to batch TS while allowing more than an order of magnitude reduction in decision-time latency. Buoyed by low latency and simplicity of implementation, our algorithm has been successfully deployed in multiple video upload systems for Meta. Using a randomized controlled trial, we show our algorithm resulted in significant improvements in video quality and watch time.