mlsanityData to 14 Oct 2026

Towards Provable Emergence of In-Context Reinforcement Learning

arXiv 2509.18389 ↗First posted 22 Sep 2025cs.LGcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Typically, a modern reinforcement learning (RL) agent solves a task by updating its neural network parameters to adapt its policy to the task. Recently, it has been observed that some RL agents can solve a wide range of new out-of-distribution tasks without parameter updates after pretraining on some task distribution. When evaluated in a new task, instead of making parameter updates, the pretrained agent conditions its policy on additional input called the context, e.g., the agent's interaction history in the new task. The agent's performance increases as the information in the context increases, with the agent's parameters fixed. This phenomenon is typically called in-context RL (ICRL). The pretrained parameters of the agent network enable the remarkable ICRL phenomenon. However, many ICRL works perform the pretraining with standard RL algorithms. This raises the central question this paper aims to address: Why can the RL pretraining algorithm generate network parameters that enable ICRL? We hypothesize that the parameters capable of ICRL are minimizers of the pretraining loss. This work provides initial support for this hypothesis through a case study. In particular, we prove that when a Transformer is pretrained for policy evaluation, one of the global minimizers of the pretraining loss can enable in-context temporal difference learning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 49 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

33 of 49 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025A Survey of In-Context Reinforcement Learning11 Feb
  2. 2025Vintix: Action Model via In-Context Reinforcement Learning31 Jan
  3. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers17 Nov
  4. 2024N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs4 Nov
  5. 2024Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling31 May
  6. 2024Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning22 May
  7. 2024In-context Exploration-Exploitation for Reinforcement Learning11 Mar

Other references · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  7. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  8. 2017A Tutorial on Thompson Samplingcited by 433
  9. 2016Learning to reinforcement learncited by 377
  10. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  12. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
14 more
  1. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  2. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  3. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  4. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  5. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  6. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  7. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  8. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 35
  9. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  10. 2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 27
  11. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 26
  12. 2021Introducing Symmetries to Black Box Meta Reinforcement Learningcited by 25
  13. 2023Cross-Episodic Curriculum for Transformer Agentscited by 10
  14. 2025Yes, Q-learning Helps Offline In-Context RLcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Safe In-Context Reinforcement Learningcited by 3
  2. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 1
  3. 2026Provable Transformer Context Compression for In-Context Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026In-Context Reinforcement Learning under Non-Stationarity: A Survey
  2. 2025In-Context Reinforcement Learning via Communicative World Models
  3. 2026Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought
  4. 2024Random Policy Enables In-Context Reinforcement Learning within Trust Horizons
  5. 2026Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context Reinforcement Learning
  6. 2026Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement
  7. 2022Large Language Models can Implement Policy Iteration
  8. 2025RL + Transformer = A General-Purpose Problem Solver