mlsanityData to 14 Oct 2026

Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning

arXiv 2405.13861 ↗First posted 22 May 2024cs.LGcited by 23

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Traditionally, reinforcement learning (RL) agents learn to solve new tasks by updating their neural network parameters through interactions with the task environment. However, recent works demonstrate that some RL agents, after certain pretraining procedures, can learn to solve unseen new tasks without parameter updates, a phenomenon known as in-context reinforcement learning (ICRL). The empirical success of ICRL is widely attributed to the hypothesis that the forward pass of the pretrained agent neural network implements an RL algorithm. In this paper, we support this hypothesis by showing, both empirically and theoretically, that when a transformer is trained for policy evaluation tasks, it can discover and learn to implement temporal difference learning in its forward pass.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. First posted on arXiv 22 May 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 10.

23 papers.

Sort:

By area:

  1. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 31
  2. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 14
  3. 2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
  4. 2026Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Datacited by 6
  5. 2024Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Featurescited by 5
  6. 2025In-Context Reinforcement Learning via Communicative World Modelscited by 5
  7. 2025Experience Replay Addresses Loss of Plasticity in Continual Learningcited by 4
  8. 2024Mental Modeling of Reinforcement Learning Agents by Language Modelscited by 3
  9. 2025Towards Provable Emergence of In-Context Reinforcement Learningcited by 3
  10. 2025Safe In-Context Reinforcement Learningcited by 3
  11. 2026Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvementcited by 3
  12. 2026Continuous Latent Contexts Enable Efficient Online Learning in Transformerscited by 3
11 more
  1. 2024Random Policy Enables In-Context Reinforcement Learning within Trust Horizonscited by 2
  2. 2026Provable and Practical In-Context Policy Optimization for Self-Improvementcited by 2
  3. 2025Linear Transformers Implicitly Discover Unified Numerical Algorithmscited by 1
  4. 2025In-Context Compositional Q-Learning for Offline Reinforcement Learningcited by 1
  5. 2025Prompt-Driven Domain Adaptation for End-to-End Autonomous Driving via In-Context RLcited by 1
  6. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 1
  7. 2026In-Context Reinforcement Learning under Non-Stationarity: A Surveycited by 1
  8. 2026medR: Reward Engineering for Clinical Offline Reinforcement Learning via Tri-Drive Potential Functionscited by 0
  9. 2026UniVAD v2: Unified Visual Anomaly Detection via Support-Conditioned Boundary Constructioncited by 0
  10. 2026From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learningcited by 0
  11. 2026PolicyAttention: Softmax Attention Implements Policy Mirror Descent for Closed-Loop Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RL + Transformer = A General-Purpose Problem Solver
  2. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining
  3. 2026One for All: A Non-Linear Transformer can Enable Cross-Domain Generalization for In-Context Reinforcement Learning
  4. 2025A Survey of In-Context Reinforcement Learning
  5. 2026In-Context Reinforcement Learning From Suboptimal Historical Data
  6. 2023Transformers in Reinforcement Learning: A Survey
  7. 2026Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning
  8. 2026Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformers