mlsanityData to 7 Oct 2026

RL2: Fast Reinforcement Learning via Slow Reinforcement Learning

arXiv 1611.02779 ↗First posted 9 Nov 2016cs.AI cs.LG cs.NEcited by 490

Not on the map. Why?

Cited by 40 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a "fast" reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL2, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose ("slow") RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the "fast" RL algorithm on the current (previously unseen) MDP. We evaluate RL2 experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL2 is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL2 on a vision-based navigation task and show that it scales up to high-dimensional problems.

Key facts. Cited by 490 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 4; 2020: 4; 2021: 93; 2022: 88; 2023: 87; 2024: 67; 2025: 80; 2026: 65.

490 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2024Automated Design of Agentic Systemscited by 360
  3. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  4. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  5. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  6. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  7. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  8. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  9. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  10. 2021Advances and Challenges in Conversational Recommender Systems: A Surveycited by 118
  11. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  12. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
38 more
  1. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  2. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  3. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  5. 2022Discovered Policy Optimisationcited by 103
  6. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  7. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  8. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  9. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  10. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  11. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  12. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  13. 2025Kevin: Multi-Turn RL for Generating CUDA Kernelscited by 73
  14. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  15. 2025Self-Adapting Language Modelscited by 71
  16. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 70
  17. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  18. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  19. 2022A Survey on Model-based Reinforcement Learningcited by 65
  20. 2021Meta-Thompson Samplingcited by 62
  21. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 61
  22. 2023Meta-in-context learning in large language modelscited by 59
  23. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  24. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  25. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  26. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  27. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  28. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
  29. 2021Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learningcited by 49
  30. 2021Evolving Reinforcement Learning Algorithmscited by 48
  31. 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 46
  32. 2021Bayesian decision-making under misspecified priors with applications to meta-learningcited by 43
  33. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  34. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
  35. 2022Transformers are Meta-Reinforcement Learnerscited by 39
  36. 2022Hypernetworks in Meta-Reinforcement Learningcited by 39
  37. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  38. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37

50 most cited of 490 All 490 (JSON)