# Thompson Sampling via Fine-Tuning of LLMs

arXiv: https://arxiv.org/abs/2510.13328 · HTML: https://mlsanity.com/paper/2510.13328/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 63 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [On Kernelized Multi-armed Bandits](https://mlsanity.com/paper/1704.00445/) (2017)
- [Learning to Optimize Via Posterior Sampling](https://mlsanity.com/paper/1301.2609/) (2013)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [Bayesian Optimization in a Billion Dimensions via Random Embeddings](https://mlsanity.com/paper/1301.1942/) (2013)
- [An Information-Theoretic Analysis of Thompson Sampling](https://mlsanity.com/paper/1403.5341/) (2014)
- [Batched Large-scale Bayesian Optimization in High-dimensional Spaces](https://mlsanity.com/paper/1706.01445/) (2017)
- [More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize](https://mlsanity.com/paper/2203.06176/) (2022)
- [Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical Space](https://mlsanity.com/paper/1706.01825/) (2017)
- [Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspaces](https://mlsanity.com/paper/1902.03229/) (2019)
- [No-Regret Bayesian Optimization with Unknown Hyperparameters](https://mlsanity.com/paper/1901.03357/) (2019)
- [Improving black-box optimization in VAE latent space using decoder uncertainty](https://mlsanity.com/paper/2107.00096/) (2021)
- [A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?](https://mlsanity.com/paper/2402.05015/) (2024)
- [Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models](https://mlsanity.com/paper/2406.14712/) (2024)
- [Diversified Sampling for Batched Bayesian Optimization with Determinantal Point Processes](https://mlsanity.com/paper/2110.11665/) (2021)
- [Latent Bayesian Optimization via Autoregressive Normalizing Flows](https://mlsanity.com/paper/2504.14889/) (2025)
- [Probabilistic Inference in Reinforcement Learning Done Right](https://mlsanity.com/paper/2311.13294/) (2023)
- [Variational Bayesian Optimistic Sampling](https://mlsanity.com/paper/2110.15688/) (2021)
- [Simplifying Bayesian Optimization Via In-Context Direct Optimum Sampling](https://mlsanity.com/paper/2505.23913/) (2025)
- [Optimistic Games for Combinatorial Bayesian Optimization with Application to Protein Design](https://mlsanity.com/paper/2409.18582/) (2024)
- [LITE: Efficiently Estimating Gaussian Probability of Maximality](https://mlsanity.com/paper/2501.13535/) (2025)

## Built on by (most cited first, 6 of 6)

- [LABO: LLM-Accelerated Bayesian Optimization through Broad Exploration and Selective Experimentation](https://mlsanity.com/paper/2605.22054/) (2026)
- [A Theoretical Analysis of Test-Driven Code Generation](https://mlsanity.com/paper/2602.06098/) (2026)
- [Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH](https://mlsanity.com/paper/2601.12355/) (2026)
- [POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensembles](https://mlsanity.com/paper/2605.07775/) (2026)
- [Agentic Search Spaces for Tabular Machine Learning](https://mlsanity.com/paper/2609.16309/) (2026)
- [Feedback-Calibrated Protein Optimization with Batch-Aligned Tail Arbitration](https://mlsanity.com/paper/2609.37808/) (2026)

## Abstract (arXiv)

Bayesian optimization in large unstructured discrete spaces is often hindered by the computational cost of maximizing acquisition functions due to the absence of gradients. We propose a scalable alternative based on Thompson sampling that eliminates the need for acquisition function maximization by directly parameterizing the probability that a candidate yields the maximum reward. Our approach, Thompson Sampling via Fine-Tuning (ToSFiT) leverages the prior knowledge embedded in prompt-conditioned large language models, and incrementally adapts them toward the posterior. Theoretically, we derive a novel regret bound for a variational formulation of Thompson Sampling that matches the strong guarantees of its standard counterpart. Our analysis reveals the critical role of careful adaptation to the posterior probability of maximality -- a principle that underpins our ToSFiT algorithm. Empirically, we validate our method on three diverse tasks: FAQ response refinement, thermally stable protein search, and quantum circuit design. Within a collection of methods covering in-context Bayesian optimization, reinforcement learning, and evolutionary search, ToSFiT exhibits both state-of-the-art sample efficiency and computational efficiency.