mlsanityData to 7 Oct 2026

In-context Reinforcement Learning with Algorithm Distillation

arXiv 2210.14215 ↗First posted 25 Oct 2022cs.LG cs.AIcited by 186

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose Algorithm Distillation (AD), a method for distilling reinforcement learning (RL) algorithms into neural networks by modeling their training histories with a causal sequence model. Algorithm Distillation treats learning to reinforcement learn as an across-episode sequential prediction problem. A dataset of learning histories is generated by a source RL algorithm, and then a causal transformer is trained by autoregressively predicting actions given their preceding learning histories as context. Unlike sequential policy prediction architectures that distill post-learning or expert sequences, AD is able to improve its policy entirely in-context without updating its network parameters. We demonstrate that AD can reinforcement learn in-context in a variety of environments with sparse rewards, combinatorial task structure, and pixel-based observations, and find that AD learns a more data-efficient RL algorithm than the one that generated the source data.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 25 Oct 2022.

Builds on

32 of 47 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
20 more
  1. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  2. 2019When Does Label Smoothing Help?cited by 985
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  4. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  5. 2015Policy Distillationcited by 414
  6. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  7. 2016Learning to reinforcement learncited by 369
  8. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  9. 2022Online Decision Transformercited by 194
  10. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  11. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  12. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  13. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  14. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  15. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  16. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  17. 2019Meta-Q-Learningcited by 62
  18. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  19. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  20. 2021Bootstrapped Meta-Learningcited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 40; 2024: 44; 2025: 52; 2026: 47.

186 papers.

Sort:

By area:

  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  3. 2023Language Modeling Is Compressioncited by 225
  4. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  5. 2023Large Language Models as General Pattern Machinescited by 196
  6. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  7. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  8. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  9. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  10. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  11. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  12. 2023The Learnability of In-Context Learningcited by 126
38 more
  1. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  2. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  3. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  4. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 89
  5. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  6. 2024Can large language models explore in-context?cited by 75
  7. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  8. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  9. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  10. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  11. 2024ICLR: In-Context Learning of Representationscited by 49
  12. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  13. 2023Large Language Models Are Semi-Parametric Reinforcement Learning Agentscited by 42
  14. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  15. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  16. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  17. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  18. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  19. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  20. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 33
  21. 2024The broader spectrum of in-context learningcited by 33
  22. 2025TESS 2: A Large-Scale Generalist Diffusion Language Modelcited by 30
  23. 2025Training a Generally Curious Agentcited by 29
  24. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 29
  25. 2023Introspective Tips: Large Language Model for In-Context Decision Makingcited by 28
  26. 2023Learning to Modulate pre-trained Models in RLcited by 28
  27. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  28. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 27
  29. 2023Teaching Language Models to Self-Improve through Interactive Demonstrationscited by 26
  30. 2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 26
  31. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 25
  32. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  33. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 23
  34. 2023Passive learning of active causal strategies in agents and language modelscited by 21
  35. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  36. 2025LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacracited by 20
  37. 2025Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learningcited by 19
  38. 2023A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Taskscited by 17

50 most cited of 186 All 186 (JSON)