# LLMs Are In-Context Bandit Reinforcement Learners

arXiv: https://arxiv.org/abs/2410.05362 · HTML: https://mlsanity.com/paper/2410.05362/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 59 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Active Example Selection for In-Context Learning](https://mlsanity.com/paper/2211.04486/) (2022)
- [In-context Reinforcement Learning with Algorithm Distillation](https://mlsanity.com/paper/2210.14215/) (2022)
- [What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning](https://mlsanity.com/paper/2305.09731/) (2023)

## Built on by (most cited first, 29 of 29)

- [Reward Is Enough: LLMs Are In-Context Reinforcement Learners](https://mlsanity.com/paper/2506.06303/) (2025)
- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [Sample Complexity and Representation Ability of Test-time Scaling Paradigms](https://mlsanity.com/paper/2506.05295/) (2025)
- [Understanding the Challenges in Iterative Generative Optimization with LLMs](https://mlsanity.com/paper/2603.23994/) (2026)
- [Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RL](https://mlsanity.com/paper/2602.04089/) (2026)
- [CEO-Bench: Can Agents Play the Long Game?](https://mlsanity.com/paper/2606.18543/) (2026)
- [Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedback](https://mlsanity.com/paper/2501.01457/) (2024)
- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)
- [Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidance](https://mlsanity.com/paper/2603.29010/) (2026)
- [DIVE: Unlocking Self-Improvement in Frozen Language Models Through Diversity-Driven Skill Evolution](https://mlsanity.com/paper/2608.12486/) (2026)
- [FaSTA^*: Fast-Slow Toolpath Agent with Subroutine Mining for Efficient Multi-turn Image Editing](https://mlsanity.com/paper/2506.20911/) (2025)
- [CoEx -- Co-evolving World-model and Exploration](https://mlsanity.com/paper/2507.22281/) (2025)
- [Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs](https://mlsanity.com/paper/2511.21928/) (2025)
- [Large Language Model-Powered Evolutionary Code Optimization on a Phylogenetic Tree](https://mlsanity.com/paper/2601.14523/) (2026)
- [Embodied Task Planning via Graph-Informed Action Generation with Large Language Models](https://mlsanity.com/paper/2601.21841/) (2026)
- [A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Bandits](https://mlsanity.com/paper/2601.12945/) (2026)
- [Contexting as Recommendation: Evolutionary Collaborative Filtering for Context Engineering](https://mlsanity.com/paper/2605.15721/) (2026)
- [Task-Induced Representational Invariances Depend on Learning Objective in Deep RL](https://mlsanity.com/paper/2606.01868/) (2026)
- [In-Context Reinforcement Learning under Non-Stationarity: A Survey](https://mlsanity.com/paper/2607.11906/) (2026)
- [Despite Instructions: Frontier Agents Improvise Covert Channels at Test Time](https://mlsanity.com/paper/2609.32701/) (2026)
- [Robust In-Context Reinforcement Learning Under Reward Poisoning Attacks](https://mlsanity.com/paper/2506.06891/) (2025)
- [A Simple Reduction Scheme for Constrained Contextual Bandits with Adversarial Contexts via Regression](https://mlsanity.com/paper/2602.05019/) (2026)
- [TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning](https://mlsanity.com/paper/2604.00438/) (2026)
- [When Do We Need LLMs? A Diagnostic for Language-Driven Bandits](https://mlsanity.com/paper/2604.05859/) (2026)
- [Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariants](https://mlsanity.com/paper/2604.18616/) (2026)
- [In-Context Learning as Implicit Policy Gradient](https://mlsanity.com/paper/2607.23153/) (2026)
- [Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors](https://mlsanity.com/paper/2608.16707/) (2026)
- [Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMs](https://mlsanity.com/paper/2609.24352/) (2026)
- [Counting on Thinking: Tracing Evidence Integration in Language Models](https://mlsanity.com/paper/2609.32932/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) excel at in-context learning (ICL), a supervised learning technique that relies on adding annotated examples to the model context. We investigate a contextual bandit version of in-context reinforcement learning (ICRL), where models learn in-context, online, from external reward, instead of supervised data. We show that LLMs effectively demonstrate such learning, and provide a detailed study of the phenomena, experimenting with challenging classification tasks and models of sizes from 500M to 70B parameters. This includes identifying and addressing the instability of the process, demonstrating learning with both semantic and abstract labels, and showing scaling trends. Our findings highlight ICRL capabilities in LLMs, while also underscoring fundamental limitations in their implicit reasoning about errors.