Data to 7 Oct 2026

Supervised Pretraining Can Learn In-Context Reinforcement Learning

arXiv 2306.14892 ↗First posted 26 Jun 2023cs.LG cs.AIcited by 137

Not on the map. Why?

Abstract (arXiv)

Large transformer models trained on diverse datasets have shown a remarkable ability to learn in-context, achieving high few-shot performance on tasks they were not explicitly trained to solve. In this paper, we study the in-context learning capabilities of transformers in decision-making problems, i.e., reinforcement learning (RL) for bandits and Markov decision processes. To do so, we introduce and study Decision-Pretrained Transformer (DPT), a supervised pretraining method where the transformer predicts an optimal action given a query state and an in-context dataset of interactions, across a diverse set of tasks. This procedure, while simple, produces a model with several surprising capabilities. We find that the pretrained transformer can be used to solve a range of RL problems in-context, exhibiting both exploration online and conservatism offline, despite not being explicitly trained to do so. The model also generalizes beyond the pretraining distribution to new tasks and automatically adapts its decision-making strategies to unknown structure. Theoretically, we show DPT can be viewed as an efficient implementation of Bayesian posterior sampling, a provably sample-efficient RL algorithm. We further leverage this connection to provide guarantees on the regret of the in-context algorithm yielded by DPT, and prove that it can learn faster than algorithms used to generate the pretraining data. These results suggest a promising yet simple path towards instilling strong in-context decision-making abilities in transformers.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 70 of its 94 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

70 of 94 references.

References · 70

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2017Proximal Policy Optimization Algorithmscited by 14,441
2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413

38 more

2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2022In-context Learning and Induction Headscited by 930
2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2022A Generalist Agentcited by 802
2022Transformers learn in-context by gradient descentcited by 714
2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
2019Behavior Regularized Offline Reinforcement Learningcited by 654
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2016Deep Exploration via Bootstrapped DQNcited by 518
2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
2017A Tutorial on Thompson Samplingcited by 420
2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
2016Learning to reinforcement learncited by 369
2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
2020Is Pessimism Provably Efficient for Offline RL?cited by 362
2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
2022Behavior Transformers: Cloning k modes with one stonecited by 299
2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
2013Learning to Optimize Via Posterior Samplingcited by 253
2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
2022Multi-Game Decision Transformerscited by 212
2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
2023The Learnability of In-Context Learningcited by 126
2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 43; 2025: 50; 2026: 30.

137 papers.

Sort:

By area:

2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
2023Large Language Models as General Pattern Machinescited by 196
2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
2024Can large language models explore in-context?cited by 75
2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
2024ICLR: In-Context Learning of Representationscited by 49
2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39

38 more

2024Evidence of Learned Look-Ahead in a Chess-Playing Neural Networkcited by 33
2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 33
2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
2025Foundation Models for Causal Inference via Prior-Data Fitted Networkscited by 30
2025Training a Generally Curious Agentcited by 29
2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 29
2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 27
2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 26
2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 25
2023In-Context Reinforcement Learning for Variable Action Spacescited by 23
2023Can a Transformer Represent a Kalman Filter?cited by 20
2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
2024Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modelingcited by 18
2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Explorationcited by 18
2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Taskscited by 17
2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 16
2024Efficient Sequential Decision Making with Large Language Modelscited by 16
2025Horizon Generalization in Reinforcement Learningcited by 15
2025Toward Efficient Exploration by Large Language Model Agentscited by 15
2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
2024Medical Vision Generalist: Unifying Medical Imaging Tasks in Contextcited by 13
2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 13
2024REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environmentscited by 12
2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planningcited by 12
2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
2023Cross-Episodic Curriculum for Transformer Agentscited by 10
2024What is it for a Machine Learning Model to Have a Capability?cited by 10
2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 10
2023Transformers are Provably Optimal In-context Estimators for Wireless Communicationscited by 8
2024Personalized Adaptation via In-Context Preference Learningcited by 8
2025Vintix: Action Model via In-Context Reinforcement Learningcited by 8
2025Should You Use Your Large Language Model to Explore or Exploit?cited by 8
2025Yes, Q-learning Helps Offline In-Context RLcited by 8

50 most cited of 137 All 137 (JSON)