mlsanityData to 7 Oct 2026

Human-Timescale Adaptation in an Open-Ended Task Space

arXiv 2301.07608 ↗First posted 18 Jan 2023cs.LG cs.AI cs.NEcited by 136

Not on the map. Why?

Abstract (arXiv)

Foundation models have shown impressive adaptation and scalability in supervised and self-supervised learning problems, but so far these successes have not fully translated to reinforcement learning (RL). In this work, we demonstrate that training an RL agent at scale leads to a general in-context learning algorithm that can adapt to open-ended novel embodied 3D problems as quickly as humans. In a vast space of held-out environment dynamics, our adaptive agent (AdA) displays on-the-fly hypothesis-driven exploration, efficient exploitation of acquired knowledge, and can successfully be prompted with first-person demonstrations. Adaptation emerges from three ingredients: (1) meta-reinforcement learning across a vast, smooth and diverse task distribution, (2) a policy parameterised as a large-scale attention-based memory architecture, and (3) an effective automated curriculum that prioritises tasks at the frontier of an agent's capabilities. We demonstrate characteristic scaling laws with respect to network size, memory length, and richness of the training task distribution. We believe our results lay the foundation for increasingly general and adaptive RL agents that perform well across ever-larger open-ended domains.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Builds on 106 of its 134 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

106 of 134 references.

References · 106

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2020GLU Variants Improve Transformercited by 1,724
38 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2021Scaling Vision Transformerscited by 993
  7. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  8. 2022A Generalist Agentcited by 802
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  10. 2019Solving Rubik's Cube with a Robot Handcited by 699
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  12. 2016Learning to learn by gradient descent by gradient descentcited by 616
  13. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  14. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  15. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  16. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  17. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  18. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  19. 2016Learning to reinforcement learncited by 369
  20. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  21. 2017Population Based Training of Neural Networkscited by 351
  22. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  23. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  24. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  25. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  26. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  27. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  28. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  29. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  30. 2018Probabilistic Model-Agnostic Meta-Learningcited by 254
  31. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  32. 2022Multi-Game Decision Transformerscited by 212
  33. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  34. 2014Robots that can adapt like animalscited by 203
  35. 2017Teacher-Student Curriculum Learningcited by 197
  36. 2020The NetHack Learning Environmentcited by 196
  37. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  38. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 44; 2024: 45; 2025: 30; 2026: 17.

136 papers.

Sort:

By area:

  1. 2024Genie: Generative Interactive Environmentscited by 706
  2. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  3. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  4. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  5. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  6. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  7. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 111
  8. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  9. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  10. 2023Synthetic Experience Replaycited by 100
  11. 2023CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalizationcited by 88
  12. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
38 more
  1. 2024Visibility into AI Agentscited by 77
  2. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  3. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  4. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  5. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  6. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  7. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  8. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  9. 2023Scaling laws for single-agent reinforcement learningcited by 39
  10. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  11. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  12. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  13. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  14. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  15. 2024The broader spectrum of in-context learningcited by 33
  16. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  17. 2023Augmenting Autotelic Agents with Large Language Modelscited by 29
  18. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 29
  19. 2023Introspective Tips: Large Language Model for In-Context Decision Makingcited by 28
  20. 2024Taking AI Welfare Seriouslycited by 28
  21. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  22. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  23. 2025Assessing Adaptive World Models in Machines with Novel Gamescited by 20
  24. 2025Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesiscited by 19
  25. 2023Recurrent Action Transformer with Memorycited by 17
  26. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  27. 2024DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraftcited by 17
  28. 2025Meta-RL Induces Exploration in Language Agentscited by 17
  29. 2023Curious Replay for Model-based Adaptationcited by 16
  30. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  31. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  32. 2023A Definition of Open-Ended Learning Problems for Goal-Conditioned Agentscited by 16
  33. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  34. 2024In-context Exploration-Exploitation for Reinforcement Learningcited by 16
  35. 2023Memory-Based Meta-Learning on Non-Stationary Distributionscited by 14
  36. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  37. 2025LocoFormer: Generalist Locomotion via Long-context Adaptationcited by 14
  38. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 13

50 most cited of 136 All 136 (JSON)