mlsanityData to 7 Oct 2026

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining

arXiv 2310.08566 ↗First posted 12 Oct 2023cs.LG cs.AI cs.CLcited by 82

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Large transformer models pretrained on offline reinforcement learning datasets have demonstrated remarkable in-context reinforcement learning (ICRL) capabilities, where they can make good decisions when prompted with interaction trajectories from unseen environments. However, when and how transformers can be trained to perform ICRL have not been theoretically well-understood. In particular, it is unclear which reinforcement-learning algorithms transformers can perform in context, and how distribution mismatch in offline training data affects the learned algorithms. This paper provides a theoretical framework that analyzes supervised pretraining for ICRL. This includes two recently proposed training methods -- algorithm distillation and decision-pretrained transformers. First, assuming model realizability, we prove the supervised-pretrained transformer will imitate the conditional expectation of the expert algorithm given the observed trajectory. The generalization error will scale with model capacity and a distribution divergence factor between the expert and offline algorithms. Second, we show transformers with ReLU attention can efficiently approximate near-optimal online reinforcement learning algorithms like LinUCB and Thompson sampling for stochastic linear bandits, and UCB-VI for tabular Markov decision processes. This provides the first quantitative analysis of the ICRL capabilities of transformers pretrained from offline trajectories.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 84 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

55 of 84 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Dichotomy of Control: Separating What You Can Control from What You Cannot24 Oct
  2. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments31 May
  3. 2022Multi-Game Decision Transformers30 May
  4. 2021Offline Meta-Reinforcement Learning with Online Self-Supervision8 Jul
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun
  7. 2020Offline Meta-Reinforcement Learning with Advantage Weighting13 Aug

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2022A Generalist Agentcited by 802
  9. 2022Transformers learn in-context by gradient descentcited by 714
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  12. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
36 more
  1. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  2. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  3. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  4. 2016Learning to reinforcement learncited by 369
  5. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  6. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  7. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  9. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  10. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  11. 2022Transformers Learn Shortcuts to Automatacited by 260
  12. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  13. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  14. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  15. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  16. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  17. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  18. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  19. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
  20. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  21. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  22. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  23. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  24. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  25. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  26. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  27. 2019Meta reinforcement learning as task inferencecited by 75
  28. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  29. 2023A Study on ReLU and Softmax in Transformercited by 53
  30. 2023Replacing softmax with ReLU in Vision Transformerscited by 43
  31. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  32. 2023Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Modelscited by 39
  33. 2015One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priorscited by 26
  34. 2020On the Global Optimality of Model-Agnostic Meta-Learningcited by 21
  35. 2021Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimallycited by 12
  36. 2022Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resetscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 32; 2025: 29; 2026: 17.

82 papers.

Sort:

By area:

  1. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  2. 2024Can large language models explore in-context?cited by 75
  3. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  4. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  5. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  6. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 33
  7. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
  8. 2025Training a Generally Curious Agentcited by 29
  9. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  10. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 23
  11. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  12. 2023Can a Transformer Represent a Kalman Filter?cited by 20
38 more
  1. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  2. 2025How Do LLMs Perform Two-Hop Reasoning in Context?cited by 20
  3. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  4. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  5. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 16
  6. 2024On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixturescited by 15
  7. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 13
  8. 2024Non-asymptotic Convergence of Training Transformers for Next-token Predictioncited by 13
  9. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
  10. 2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12
  11. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  12. 2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
  13. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignmentcited by 9
  14. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  15. 2025Should You Use Your Large Language Model to Explore or Exploit?cited by 8
  16. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  17. 2024Automatic Domain Adaptation by Transformers in In-Context Learningcited by 7
  18. 2025On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learningcited by 7
  19. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  20. 2025Adaptive Elicitation of Latent Information Using Natural Languagecited by 7
  21. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 7
  22. 2026Efficient and Minimax Optimal In-context Nonparametric Regression with Transformerscited by 7
  23. 2024Benchmarking General-Purpose In-Context Learningcited by 6
  24. 2024Context-Scaling versus Task-Scaling in In-Context Learningcited by 6
  25. 2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inferencecited by 6
  26. 2026Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RLcited by 6
  27. 2023AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learningcited by 5
  28. 2024Task Diversity Shortens the ICL Plateaucited by 5
  29. 2024Global Convergence in Training Large-Scale Transformerscited by 5
  30. 2025Scalable In-Context Q-Learningcited by 5
  31. 2025Interaction as Intelligence: Deep Research With Human-AI Partnershipcited by 5
  32. 2026Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Datacited by 5
  33. 2024Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Makingcited by 4
  34. 2025On the Robustness of Transformers against Context Hijacking for Linear Classificationcited by 4
  35. 2025Learning-To-Measure: In-Context Active Feature Acquisitioncited by 4
  36. 2024Neural Networks Generalize on Low Complexity Datacited by 3
  37. 2024Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidancecited by 3
  38. 2025Filtering Learning Histories Enhances In-Context Reinforcement Learningcited by 3

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026In-Context Reinforcement Learning From Suboptimal Historical Data
  2. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner
  3. 2026Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformers
  4. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
  5. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?
  6. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  7. 2026In-Context Reinforcement Learning under Non-Stationarity: A Survey
  8. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer