# LLMs-augmented Contextual Bandit

arXiv: https://arxiv.org/abs/2311.02268 · HTML: https://mlsanity.com/paper/2311.02268/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Introduction to Multi-Armed Bandits](https://mlsanity.com/paper/1904.07272/) (2019)
- [Practical Contextual Bandits with Regression Oracles](https://mlsanity.com/paper/1803.01088/) (2018)
- [Federated Linear Contextual Bandits](https://mlsanity.com/paper/2110.14177/) (2021)
- [Neural Contextual Bandits without Regret](https://mlsanity.com/paper/2107.03144/) (2021)
- [A Neural Networks Committee for the Contextual Bandit Problem](https://mlsanity.com/paper/1409.8191/) (2014)
- [Random Forest for the Contextual Bandit Problem - extended version](https://mlsanity.com/paper/1504.06952/) (2015)
- [Contextual Decision Trees](https://mlsanity.com/paper/2207.06355/) (2022)

## Built on by (most cited first, 8 of 8)

- [Survey: Multi-Armed Bandits Meet Large Language Models](https://mlsanity.com/paper/2505.13355/) (2025)
- [Where Does Reasoning Break? Step-Level Hallucination Detection via Hidden-State Transport Geometry](https://mlsanity.com/paper/2605.13772/) (2026)
- [Hierarchical Upper Confidence Bounds for Constrained Online Learning](https://mlsanity.com/paper/2410.17216/) (2024)
- [A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Bandits](https://mlsanity.com/paper/2601.12945/) (2026)
- [Calibration-Gated LLM Pseudo-Observations for Online Contextual Bandits](https://mlsanity.com/paper/2604.14961/) (2026)
- [Flow-Corrected Thompson Sampling for Non-Stationary Contextual Bandits](https://mlsanity.com/paper/2606.23933/) (2026)
- [Dynamic Governance of Multi-LLM Agent Systems for Collaborative Conversational Outcomes](https://mlsanity.com/paper/2608.11207/) (2026)
- [Safety by Design: Realized-Cost Constraints for Contextual Bandits with Continuous Actions](https://mlsanity.com/paper/2608.26755/) (2026)

## Abstract (arXiv)

Contextual bandits have emerged as a cornerstone in reinforcement learning, enabling systems to make decisions with partial feedback. However, as contexts grow in complexity, traditional bandit algorithms can face challenges in adequately capturing and utilizing such contexts. In this paper, we propose a novel integration of large language models (LLMs) with the contextual bandit framework. By leveraging LLMs as an encoder, we enrich the representation of the context, providing the bandit with a denser and more informative view. Preliminary results on synthetic datasets demonstrate the potential of this approach, showing notable improvements in cumulative rewards and reductions in regret compared to traditional bandit algorithms. This integration not only showcases the capabilities of LLMs in reinforcement learning but also opens the door to a new era of contextually-aware decision systems.