mlsanityData to 14 Oct 2026

Generalization to New Sequential Decision Making Tasks with In-Context Learning

arXiv 2312.03801 ↗First posted 6 Dec 2023cs.LG cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

Training autonomous agents that can learn new tasks from only a handful of demonstrations is a long-standing problem in machine learning. Recently, transformers have been shown to learn new language or vision tasks without any weight updates from only a few examples, also referred to as in-context learning. However, the sequential decision making setting poses additional challenges having a lower tolerance for errors since the environment's stochasticity or the agent's actions can lead to unseen, and sometimes unrecoverable, states. In this paper, we use an illustrative example to show that naively applying transformers to sequential decision making problems does not enable in-context learning of new tasks. We then demonstrate how training on sequences of trajectories with certain distributional properties leads to in-context learning of new sequential decision making tasks. We investigate different design choices and find that larger model and dataset sizes, as well as more task diversity, environment stochasticity, and trajectory burstiness, all result in better in-context learning of new out-of-distribution tasks. By training on large diverse offline datasets, our model is able to learn new MiniHack and Procgen tasks without any weight updates from just a handful of demonstrations.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 55 of its 87 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

55 of 87 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  8. 2022A Generalist Agentcited by 808
  9. 2022Transformers learn in-context by gradient descentcited by 732
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
38 more
  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  3. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  5. 2020Reinforcement Learning with Augmented Datacited by 468
  6. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  7. 2022Multi-Game Decision Transformerscited by 214
  8. 2022Predictability and Surprise in Large Generative Modelscited by 210
  9. 2020The NetHack Learning Environmentcited by 198
  10. 2022Online Decision Transformercited by 196
  11. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  12. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  13. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  14. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  15. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  16. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  17. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  18. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  19. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  20. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  21. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  22. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  23. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  24. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  25. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  26. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  27. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  28. 2018Task-Embedded Control Networks for Few-Shot Imitation Learningcited by 66
  29. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 63
  30. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 53
  31. 2022Exploration via Elliptical Episodic Bonusescited by 51
  32. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 49
  33. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  34. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  35. 2022Transformers are Meta-Reinforcement Learnerscited by 41
  36. 2023Hyper-Decision Transformer for Efficient Online Policy Adaptationcited by 33
  37. 2022Token Turing Machinescited by 25
  38. 2018Backplay: "Man muss immer umkehren"cited by 21

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 18; 2025: 15; 2026: 5.

39 papers.

Sort:

By area:

  1. 2024Can large language models explore in-context?cited by 76
  2. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 45
  3. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41
  4. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
  5. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  6. 2025Training a Generally Curious Agentcited by 31
  7. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 31
  8. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 29
  9. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 24
  10. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 22
  11. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  12. 2024Efficient Sequential Decision Making with Large Language Modelscited by 16
27 more
  1. 2024REGENT: A Retrieval-Augmented Generalist Agent That Can Act In-Context in New Environmentscited by 16
  2. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 15
  3. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 14
  4. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  5. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  6. 2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12
  7. 2025Yes, Q-learning Helps Offline In-Context RLcited by 10
  8. 2024Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lenscited by 8
  9. 2025Should You Use Your Large Language Model to Explore or Exploit?cited by 8
  10. 2026ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?cited by 8
  11. 2024ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AIcited by 7
  12. 2026Scaling In-Context Online Learning Capability of LLMs via Cross-Episode Meta-RLcited by 7
  13. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 5
  14. 2025In-Context Reinforcement Learning via Communicative World Modelscited by 5
  15. 2024Read to Play (R2-Play): Decision Transformer with Multimodal Game Instructioncited by 4
  16. 2025Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worldscited by 4
  17. 2025Comparing Exploration-Exploitation Strategies of LLMs and Humans: Insights from Standard Multi-armed Bandit Experimentscited by 4
  18. 2025Safe In-Context Reinforcement Learningcited by 3
  19. 2024SplAgger: Split Aggregation for Meta-Reinforcement Learningcited by 2
  20. 2025Large Language Models Develop Novel Social Biases Through Adaptive Explorationcited by 2
  21. 2026Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamworkcited by 2
  22. 2024The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Datacited by 1
  23. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 1
  24. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
  25. 2025Meta-Learning Transformers to Improve In-Context Generalizationcited by 0
  26. 2025Context Value Informed In-Context Reinforcement Learningcited by 0
  27. 2026Towards Scalable Multi-Task Reinforcement Learning with Large Decision Modelscited by 0