mlsanityData to 14 Oct 2026

A Tutorial on Thompson Sampling

arXiv 1707.02038 ↗First posted 7 Jul 2017cs.LGcited by 433

Not on the map. Why?

Cited by 38 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Thompson sampling is an algorithm for online decision problems where actions are taken sequentially in a manner that must balance between exploiting what is known to maximize immediate performance and investing to accumulate new information that may improve future performance. The algorithm addresses a broad range of problems in a computationally efficient manner and is therefore enjoying wide use. This tutorial covers the algorithm and its application, illustrating concepts through a range of examples, including Bernoulli bandit problems, shortest path problems, product recommendation, assortment, active learning with neural networks, and reinforcement learning in Markov decision processes. Most of these problems involve complex information structures, where information revealed by taking an action informs beliefs about other actions. We will also discuss when and why Thompson sampling is or is not effective and relations to alternative algorithms.

Key facts. Cited by 433 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 63; 2022: 69; 2023: 64; 2024: 77; 2025: 69; 2026: 89.

433 papers.

Sort:

By area:

  1. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  2. 2021Epistemic Neural Networkscited by 120
  3. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 107
  4. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 79
  5. 2022Recent Advances in Bayesian Optimizationcited by 76
  6. 2024Can large language models explore in-context?cited by 76
  7. 2024Variational Bayesian Last Layerscited by 69
  8. 2021Meta-Thompson Samplingcited by 63
  9. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  10. 2021Reinforcement Learning, Bit by Bitcited by 59
  11. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  12. 2024Code Repair with LLMs gives an Exploration-Exploitation Tradeoffcited by 51
38 more
  1. 2021Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challengescited by 50
  2. 2022Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robotscited by 47
  3. 2024Efficient Exploration for LLMscited by 45
  4. 2021Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Banditscited by 40
  5. 2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 36
  6. 2021Hierarchical Bayesian Banditscited by 35
  7. 2025MAT-Agent: Adaptive Multi-Agent Training Optimizationcited by 35
  8. 2024Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimizationcited by 34
  9. 2022Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexitycited by 32
  10. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  11. 2023Thompson sampling for improved exploration in GFlowNetscited by 32
  12. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  13. 2021No Regrets for Learning the Prior in Banditscited by 30
  14. 2021From Predictions to Decisions: The Importance of Joint Predictive Distributionscited by 29
  15. 2024A Unified Confidence Sequence for Generalized Linear Models, with Applications to Banditscited by 29
  16. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Rewardcited by 28
  17. 2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 28
  18. 2022Hindsight Learning for MDPs with Exogenous Inputscited by 26
  19. 2021Metadata-based Multi-Task Bandits with Bayesian Hierarchical Modelscited by 24
  20. 2021Diversified Sampling for Batched Bayesian Optimization with Determinantal Point Processescited by 24
  21. 2022An Analysis of Ensemble Samplingcited by 24
  22. 2023Improved Regret Bounds of (Multinomial) Logistic Bandits via Regret-to-Confidence-Set Conversioncited by 24
  23. 2021Deciding What to Learn: A Rate-Distortion Approachcited by 23
  24. 2023Low-rank extended Kalman filtering for online learning of neural networks from streaming datacited by 23
  25. 2021Online Multi-Armed Bandits with Adaptive Inferencecited by 22
  26. 2021The Neural Testbed: Evaluating Joint Predictionscited by 22
  27. 2022Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelinescited by 22
  28. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  29. 2022Langevin Monte Carlo for Contextual Banditscited by 22
  30. 2021Policy Learning with Adaptively Collected Datacited by 21
  31. 2022Non-Stationary Bandit Learning via Predictive Samplingcited by 21
  32. 2021Parallelizing Thompson Samplingcited by 20
  33. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 20
  34. 2021Reinforcement Learning for Feedback-Enabled Cyber Resiliencecited by 19
  35. 2022BayesFormer: Transformer with Uncertainty Estimationcited by 19
  36. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  37. 2022Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrappingcited by 19
  38. 2022Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Carecited by 18

50 most cited of 433 All 433 (JSON)