mlsanityData to 7 Oct 2026

Contextual Thompson Sampling via Generation of Missing Data

arXiv 2502.07064 ↗First posted 10 Feb 2025cs.LG cs.AI stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce a framework for Thompson sampling (TS) contextual bandit algorithms, in which the algorithm's ability to quantify uncertainty and make decisions depends on the quality of a generative model that is learned offline. Instead of viewing uncertainty in the environment as arising from unobservable latent parameters, our algorithm treats uncertainty as stemming from missing, but potentially observable outcomes (including both future and counterfactual outcomes). If these outcomes were all observed, one could simply make decisions using an "oracle" policy fit on the complete dataset. Inspired by this conceptualization, at each decision-time, our algorithm uses a generative model to probabilistically impute missing outcomes, fits a policy using the imputed complete dataset, and uses that policy to select the next action. We formally show that this algorithm is a generative formulation of TS and establish a state-of-the-art regret bound. Notably, our regret bound depends on the generative model only through the quality of its offline prediction loss, and applies to any method of fitting the "oracle" policy.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 47 of its 81 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

47 of 81 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2020Efficient Transformers: A Surveycited by 784
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  8. 2018Conditional Neural Processescited by 421
  9. 2017A Tutorial on Thompson Samplingcited by 420
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  11. 2017Algorithmic decision making and the cost of fairnesscited by 402
  12. 2021Transformers Can Do Bayesian Inferencecited by 338
35 more
  1. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  2. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  3. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  4. 2020Bayesian Neural Networks: An Introduction and Surveycited by 130
  5. 2020Neural Thompson Samplingcited by 127
  6. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  7. 2016Corralling a Band of Bandit Algorithmscited by 99
  8. 2019Distilling Policy Distillationcited by 81
  9. 2017Ensemble Samplingcited by 62
  10. 2021Meta-Thompson Samplingcited by 62
  11. 2019Model selection for contextual banditscited by 60
  12. 2020Meta-learning with Stochastic Linear Banditscited by 45
  13. 2021Bayesian decision-making under misspecified priors with applications to meta-learningcited by 43
  14. 2015Bootstrapped Thompson Sampling and Deep Explorationcited by 40
  15. 2013Sequential Transfer in Multi-armed Bandit with Finite Set of Modelscited by 38
  16. 2020Hypermodels for Explorationcited by 37
  17. 2018An Information-Theoretic Analysis for Thompson Sampling with Many Actionscited by 35
  18. 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
  19. 2021No Regrets for Learning the Prior in Banditscited by 29
  20. 2021From Predictions to Decisions: The Importance of Joint Predictive Distributionscited by 28
  21. 2023Don't lose the message while paraphrasing: A study on content preserving style transfercited by 25
  22. 2021Metadata-based Multi-Task Bandits with Bayesian Hierarchical Modelscited by 24
  23. 2022An Analysis of Ensemble Samplingcited by 23
  24. 2022Lifting the Information Ratio: An Information-Theoretic Analysis of Thompson Sampling for Contextual Banditscited by 19
  25. 2021Thompson Sampling with a Mixture Priorcited by 15
  26. 2015On the Prior Sensitivity of Thompson Samplingcited by 14
  27. 2019Thompson Sampling via Local Uncertaintycited by 12
  28. 2020Open Problem: Model Selection for Contextual Banditscited by 12
  29. 2023An Information-Theoretic Analysis of Nonstationary Bandit Learningcited by 11
  30. 2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
  31. 2024Active Exploration via Autoregressive Generation of Missing Datacited by 8
  32. 2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapcited by 7
  33. 2024Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgentcited by 6
  34. 2022Multi-Environment Meta-Learning in Stochastic Linear Banditscited by 4
  35. 2024An Information-Theoretic Analysis of Thompson Sampling for Logistic Banditscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Pre-Trained AI Model Assisted Online Decision-Making under Missing Covariates: A Theoretical Perspectivecited by 3
  2. 2026PFN-TS: Thompson Sampling for Contextual Bandits via Prior-Data Fitted Networkscited by 2
  3. 2026OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agentscited by 2
  4. 2025Learning with Incomplete Context: Linear Contextual Bandits with Pretrained Imputationcited by 0
  5. 2026Bootstrap-Conditioned Action Selection with Tabular Foundation Modelscited by 0