# Large Language Model-Enhanced Multi-Armed Bandits

arXiv: https://arxiv.org/abs/2502.01118 · HTML: https://mlsanity.com/paper/2502.01118/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oracles](https://mlsanity.com/paper/2002.04926/) (2020)
- [In-context Reinforcement Learning with Algorithm Distillation](https://mlsanity.com/paper/2210.14215/) (2022)
- [Supervised Pretraining Can Learn In-Context Reinforcement Learning](https://mlsanity.com/paper/2306.14892/) (2023)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024)
- [Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning](https://mlsanity.com/paper/2412.09078/) (2024)
- [Practical Contextual Bandits with Regression Oracles](https://mlsanity.com/paper/1803.01088/) (2018)
- [Can large language models explore in-context?](https://mlsanity.com/paper/2403.15371/) (2024)
- [Efficient Exploration for LLMs](https://mlsanity.com/paper/2402.00396/) (2024)
- [Misspecified Linear Bandits](https://mlsanity.com/paper/1704.06880/) (2017)
- [EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration](https://mlsanity.com/paper/2410.06238/) (2024)
- [In-context Exploration-Exploitation for Reinforcement Learning](https://mlsanity.com/paper/2403.06826/) (2024)
- [Online Clustering of Bandits with Misspecified User Models](https://mlsanity.com/paper/2310.02717/) (2023)
- [Zeroth Order Non-convex optimization with Dueling-Choice Bandits](https://mlsanity.com/paper/1911.00980/) (2019)
- [Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learning](https://mlsanity.com/paper/2406.05064/) (2024)
- [Should You Use Your Large Language Model to Explore or Exploit?](https://mlsanity.com/paper/2502.00225/) (2025)
- [Neural Dueling Bandits: Preference-Based Optimization with Human Feedback](https://mlsanity.com/paper/2407.17112/) (2024)

## Built on by (most cited first, 7 of 7)

- [Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RL](https://mlsanity.com/paper/2602.04089/) (2026)
- [Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach](https://mlsanity.com/paper/2511.04393/) (2025)
- [Why Keep Your Doubts to Yourself? Trading Visual Uncertainties in Multi-Agent Bandit Systems](https://mlsanity.com/paper/2601.18735/) (2026)
- [TextBandit: Evaluating Probabilistic Reasoning in LLMs Through Language-Only Decision Tasks](https://mlsanity.com/paper/2510.13878/) (2025)
- [GRIMIP: A General Framework for Instance-Specific Configuration of MIP Solvers Using LLMs](https://mlsanity.com/paper/2606.23299/) (2026)
- [When Do We Need LLMs? A Diagnostic for Language-Driven Bandits](https://mlsanity.com/paper/2604.05859/) (2026)
- [Calibration-Gated LLM Pseudo-Observations for Online Contextual Bandits](https://mlsanity.com/paper/2604.14961/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been adopted to solve sequential decision-making tasks such as multi-armed bandits (MAB), in which an LLM is directly instructed to select the arms to pull in every iteration. However, this paradigm of direct arm selection using LLMs has been shown to be suboptimal in many MAB tasks. Therefore, we propose an alternative approach which combines the strengths of classical MAB and LLMs. Specifically, we adopt a classical MAB algorithm as the high-level framework and leverage the strong in-context learning capability of LLMs to perform the sub-task of reward prediction. Firstly, we incorporate the LLM-based reward predictor into the classical Thompson sampling (TS) algorithm and adopt a decaying schedule for the LLM temperature to ensure a transition from exploration to exploitation. Next, we incorporate the LLM-based reward predictor (with a temperature of 0) into a regression oracle-based MAB algorithm equipped with an explicit exploration mechanism. We also extend our TS-based algorithm to dueling bandits where only the preference feedback between pairs of arms is available, which requires non-trivial algorithmic modifications. We conduct empirical evaluations using both synthetic MAB tasks and experiments designed using real-world text datasets, in which the results show that our algorithms consistently outperform previous baseline methods based on direct arm selection. Interestingly, we also demonstrate that in challenging tasks where the arms lack semantic meanings that can be exploited by the LLM, our approach achieves considerably better performance than LLM-based direct arm selection.