mlsanityData to 7 Oct 2026

LLMs Are In-Context Bandit Reinforcement Learners

arXiv 2410.05362 ↗First posted 7 Oct 2024cs.CL cs.AI cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) excel at in-context learning (ICL), a supervised learning technique that relies on adding annotated examples to the model context. We investigate a contextual bandit version of in-context reinforcement learning (ICRL), where models learn in-context, online, from external reward, instead of supervised data. We show that LLMs effectively demonstrate such learning, and provide a detailed study of the phenomena, experimenting with challenging classification tasks and models of sizes from 500M to 70B parameters. This includes identifying and addressing the instability of the process, demonstrating learning with both semantic and abstract labels, and showing scaling trends. Our findings highlight ICRL capabilities in LLMs, while also underscoring fundamental limitations in their implicit reasoning about errors.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 46 of its 59 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

46 of 59 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
34 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  3. 2022Transformers learn in-context by gradient descentcited by 714
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  6. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  8. 2023Larger language models do in-context learning differentlycited by 386
  9. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  10. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  11. 2023In-Context Learning Creates Task Vectorscited by 302
  12. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  13. 2024Many-Shot In-Context Learningcited by 203
  14. 2023Large Language Models as General Pattern Machinescited by 196
  15. 2022Active Example Selection for In-Context Learningcited by 195
  16. 2022Online Decision Transformercited by 194
  17. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  18. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  19. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  20. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  21. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  22. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  23. 2024Can large language models explore in-context?cited by 75
  24. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  25. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  26. 2018A Contextual Bandit Bake-offcited by 64
  27. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  28. 2024In-Context Principle Learning from Mistakescited by 42
  29. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  30. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  31. 2022Large Language Models can Implement Policy Iterationcited by 31
  32. 2024Controlling Large Language Model Agents with Entropic Activation Steeringcited by 29
  33. 2019Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedbackcited by 21
  34. 2024What Do Language Models Learn in Context? The Structured Task Hypothesiscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 20.

27 papers.

Sort:

By area:

  1. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 29
  2. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 16
  3. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  4. 2026Understanding the Challenges in Iterative Generative Optimization with LLMscited by 8
  5. 2026Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RLcited by 6
  6. 2024Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedbackcited by 5
  7. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  8. 2026Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidancecited by 4
  9. 2026CEO-Bench: Can Agents Play the Long Game?cited by 4
  10. 2025FaSTA^*: Fast-Slow Toolpath Agent with Subroutine Mining for Efficient Multi-turn Image Editingcited by 3
  11. 2025CoEx -- Co-evolving World-model and Explorationcited by 3
  12. 2025Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMscited by 2
15 more
  1. 2026Large Language Model-Powered Evolutionary Code Optimization on a Phylogenetic Treecited by 2
  2. 2026Embodied Task Planning via Graph-Informed Action Generation with Large Language Modelscited by 2
  3. 2026DIVE: Unlocking Self-Improvement in Frozen Language Models Through Diversity-Driven Skill Evolutioncited by 2
  4. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  5. 2026Task-Induced Representational Invariances Depend on Learning Objective in Deep RLcited by 1
  6. 2026Despite Instructions: Frontier Agents Improvise Covert Channels at Test Timecited by 1
  7. 2025Robust In-Context Reinforcement Learning Under Reward Poisoning Attackscited by 0
  8. 2026A Simple Reduction Scheme for Constrained Contextual Bandits with Adversarial Contexts via Regressioncited by 0
  9. 2026TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learningcited by 0
  10. 2026When Do We Need LLMs? A Diagnostic for Language-Driven Banditscited by 0
  11. 2026Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariantscited by 0
  12. 2026In-Context Learning as Implicit Policy Gradientcited by 0
  13. 2026Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priorscited by 0
  14. 2026Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMscited by 0
  15. 2026Counting on Thinking: Tracing Evidence Integration in Language Modelscited by 0