mlsanityData to 14 Oct 2026

Transformers are Meta-Reinforcement Learners

arXiv 2206.06614 ↗First posted 14 Jun 2022cs.LG cs.AIcited by 41

Not on the map. Why?

Abstract (arXiv)

The transformer architecture and variants presented remarkable success across many machine learning tasks in recent years. This success is intrinsically related to the capability of handling long sequences and the presence of context-dependent weights from the attention mechanism. We argue that these capabilities suit the central role of a Meta-Reinforcement Learning algorithm. Indeed, a meta-RL agent needs to infer the task from a sequence of trajectories. Furthermore, it requires a fast adaptation strategy to adapt its policy for a new task -- which can be achieved using the self-attention mechanism. In this work, we present TrMRL (Transformers for Meta-Reinforcement Learning), a meta-RL agent that mimics the memory reinstatement mechanism using the transformer architecture. It associates the recent past of working memories to build an episodic memory recursively through the transformer layers. We show that the self-attention computes a consensus representation that minimizes the Bayes Risk at each layer and provides meaningful features to compute the best actions. We conducted experiments in high-dimensional continuous control environments for locomotion and dexterous manipulation. Results show that TrMRL presents comparable or superior asymptotic performance, sample efficiency, and out-of-distribution generalization compared to the baselines in these environments.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 40 of its 63 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

40 of 63 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2017Proximal Policy Optimization Algorithmscited by 15,215
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2016Layer Normalizationcited by 5,669
  10. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  11. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  12. 2015Trust Region Policy Optimizationcited by 2,580
28 more
  1. 2016Matching Networks for One Shot Learningcited by 2,255
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  5. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  7. 2020On Layer Normalization in the Transformer Architecturecited by 922
  8. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  10. 2016Learning to learn by gradient descent by gradient descentcited by 620
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  12. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  13. 2016Learning to reinforcement learncited by 377
  14. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  15. 2018Probabilistic Model-Agnostic Meta-Learningcited by 256
  16. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  17. 2016Learning to Optimizecited by 195
  18. 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 130
  19. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  20. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 81
  21. 2020Discovering Reinforcement Learning Algorithmscited by 77
  22. 2019Meta-learning of Sequential Strategiescited by 70
  23. 2021Evolving Reinforcement Learning Algorithmscited by 49
  24. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 36
  25. 2018Been There, Done That: Meta-Learning with Episodic Recallcited by 26
  26. 2019Working Memory Graphscited by 23
  27. 2020Rapid Task-Solving in Novel Environmentscited by 23
  28. 2019Bottom-Up Meta-Policy Searchcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 15; 2024: 14; 2025: 6; 2026: 3.

41 papers.

Sort:

By area:

  1. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  2. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 63
  3. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  4. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  5. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  6. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  7. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  8. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  9. 2024Zero-Shot Reinforcement Learning via Function Encoderscited by 18
  10. 2023Recurrent Action Transformer with Memorycited by 17
  11. 2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalizationcited by 16
  12. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 15
29 more
  1. 2022Sim-to-Real Transfer for Quadrupedal Locomotion via Terrain Transformercited by 14
  2. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 14
  3. 2023Unraveling the ARC Puzzle: Mimicking Human Solutions with Object-Centric Decision Transformercited by 12
  4. 2023Transformers in Reinforcement Learning: A Surveycited by 12
  5. 2024X-Light: Cross-City Traffic Signal Control Using Transformer on Transformer as Meta Multi-Agent Reinforcement Learnercited by 10
  6. 2024Hierarchical Transformers are Efficient Meta-Reinforcement Learnerscited by 8
  7. 2026Meta-Reinforcement Learning with Self-Reflection for Agentic Searchcited by 8
  8. 2024ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AIcited by 7
  9. 2022Transformer in Transformer as Backbone for Deep Reinforcement Learningcited by 6
  10. 2024DreamWaQ++: Obstacle-Aware Quadrupedal Locomotion With Resilient Multi-Modal Reinforcement Learningcited by 6
  11. 2024Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learningcited by 5
  12. 2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluationcited by 5
  13. 2025Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalizationcited by 5
  14. 2023PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Taskscited by 4
  15. 2024Enhancing IoT Intelligence: A Transformer-based Reinforcement Learning Methodologycited by 4
  16. 2023Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approachcited by 3
  17. 2024Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversitycited by 3
  18. 2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
  19. 2025End-to-End Analysis of Charge Stability Diagrams with Transformerscited by 3
  20. 2023On First-Order Meta-Reinforcement Learning with Moreau Envelopescited by 2
  21. 2023PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithmcited by 2
  22. 2022Memory in humans and deep language models: Linking hypotheses for model augmentationcited by 1
  23. 2024MarineFormer: A Spatio-Temporal Attention Model for USV Navigation in Dynamic Marine Environmentscited by 1
  24. 2025TIMRL: A Novel Meta-Reinforcement Learning Framework for Non-Stationary and Multi-Task Environmentscited by 1
  25. 2023RL3: Boosting Meta Reinforcement Learning via RL inside RL2cited by 0
  26. 2025Context Representation via Action-Free Transformer encoder-decoder for Meta Reinforcement Learningcited by 0
  27. 2025Meta-RL with Bayesian Linear Task Modelscited by 0
  28. 2026Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agentcited by 0
  29. 2026Dynamic Context Scheduling: Learning Beyond the Static Universecited by 0