# Survey: Multi-Armed Bandits Meet Large Language Models

arXiv: https://arxiv.org/abs/2505.13355 · HTML: https://mlsanity.com/paper/2505.13355/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 71 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 30 of 45)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents](https://mlsanity.com/paper/2403.02502/) (2024)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [Matching Patients to Clinical Trials with Large Language Models](https://mlsanity.com/paper/2307.15051/) (2023)
- [Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2504.12501/) (2025)
- [Learning to Generate Better Than Your LLM](https://mlsanity.com/paper/2306.11816/) (2023)
- [An ADMM Based Framework for AutoML Pipeline Configuration](https://mlsanity.com/paper/1905.00424/) (2019)
- [StreamBench: Towards Benchmarking Continuous Improvement of Language Agents](https://mlsanity.com/paper/2406.08747/) (2024)
- [Efficient Prompt Optimization Through the Lens of Best Arm Identification](https://mlsanity.com/paper/2402.09723/) (2024)
- [Hierarchical Bayesian Bandits](https://mlsanity.com/paper/2111.06929/) (2021)
- [LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing](https://mlsanity.com/paper/2502.02743/) (2025)
- [Scalable Recollections for Continual Lifelong Learning](https://mlsanity.com/paper/1711.06761/) (2017)
- [Which LLM to Play? Convergence-Aware Online Model Selection with Time-Increasing Bandits](https://mlsanity.com/paper/2403.07213/) (2024)
- [Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis](https://mlsanity.com/paper/2402.11398/) (2024)
- [Incorporating Behavioral Constraints in Online AI Systems](https://mlsanity.com/paper/1809.05720/) (2018)
- [The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation](https://mlsanity.com/paper/2501.01329/) (2025)
- [Context Attentive Bandits: Contextual Bandit with Restricted Context](https://mlsanity.com/paper/1705.03821/) (2017)
- [Generative AI for the Optimization of Next-Generation Wireless Networks: Basics, State-of-the-Art, and Open Challenges](https://mlsanity.com/paper/2405.17454/) (2024)
- [A Story of Two Streams: Reinforcement Learning Models from Human Behavior and Neuropsychiatry](https://mlsanity.com/paper/1906.11286/) (2019)
- [Adam with Bandit Sampling for Deep Learning](https://mlsanity.com/paper/2010.12986/) (2020)
- [Non-Stationary Bandits with Auto-Regressive Temporal Dependency](https://mlsanity.com/paper/2210.16386/) (2022)
- [LLMs-augmented Contextual Bandit](https://mlsanity.com/paper/2311.02268/) (2023)
- [Personalized Adaptation via In-Context Preference Learning](https://mlsanity.com/paper/2410.14001/) (2024)
- [A Multi-Agent Conversational Bandit Approach to Online Evaluation and Selection of User-Aligned LLM Responses](https://mlsanity.com/paper/2501.01849/) (2025)

## Built on by (most cited first, 7 of 7)

- [Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs](https://mlsanity.com/paper/2510.09201/) (2025)
- [When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training](https://mlsanity.com/paper/2509.24923/) (2025)
- [Bandit Learning in General Open Multi-agent Systems](https://mlsanity.com/paper/2605.06202/) (2026)
- [CUPID in the Model Zoo: Online Matchmaking for Selecting Your Dream LLM](https://mlsanity.com/paper/2606.00846/) (2026)
- [Dynamic Governance of Multi-LLM Agent Systems for Collaborative Conversational Outcomes](https://mlsanity.com/paper/2608.11207/) (2026)
- [Structure, Not Belief: Correlated Thompson Sampling from LLM-Derived Covariance in Combinatorial Semi-Bandits](https://mlsanity.com/paper/2610.07470/) (2026)
- [PRAXIS: Learning Dynamics of Self-Improving Models with Symbolic Archives](https://mlsanity.com/paper/2610.11803/) (2026)

## Abstract (arXiv)

Bandit algorithms and Large Language Models (LLMs) have emerged as powerful tools in artificial intelligence, each addressing distinct yet complementary challenges in decision-making and natural language processing. This survey explores the synergistic potential between these two fields, highlighting how bandit algorithms can enhance the performance of LLMs and how LLMs, in turn, can provide novel insights for improving bandit-based decision-making. We first examine the role of bandit algorithms in optimizing LLM fine-tuning, prompt engineering, and adaptive response generation, focusing on their ability to balance exploration and exploitation in large-scale learning tasks. Subsequently, we explore how LLMs can augment bandit algorithms through advanced contextual understanding, dynamic adaptation, and improved policy selection using natural language reasoning. By providing a comprehensive review of existing research and identifying key challenges and opportunities, this survey aims to bridge the gap between bandit algorithms and LLMs, paving the way for innovative applications and interdisciplinary research in AI.