mlsanityData to 7 Oct 2026

Stabilizing Transformers for Reinforcement Learning

arXiv 1910.06764 ↗First posted 13 Oct 2019cs.LG cs.AI stat.MLcited by 262

Not on the map. Why?

Abstract (arXiv)

Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine translation. Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting. In this work we demonstrate that the standard transformer architecture is difficult to optimize, which was previously observed in the supervised learning setting but becomes especially pronounced with RL objectives. We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant. The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture. We show that the GTrXL, trained using the same losses, has stability and performance that consistently matches or exceeds a competitive LSTM baseline, including on more reactive tasks where memory is less critical. GTrXL offers an easy-to-train, simple-to-implement but substantially more expressive architectural alternative to the standard multi-layer LSTM ubiquitously used for RL agents in partially observable environments.

Key facts. Cited by 262 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 48; 2022: 44; 2023: 44; 2024: 49; 2025: 41; 2026: 34.

262 papers.

Sort:

By area:

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  3. 2021Random Feature Attentioncited by 350
  4. 2022Transformers are Sample-Efficient World Modelscited by 317
  5. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  6. 2021Generating Images with Sparse Representationscited by 262
  7. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  8. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  9. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  10. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  11. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  12. 2023Large Language Models for Robotics: A Surveycited by 132
38 more
  1. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  2. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  3. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  4. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  5. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  6. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  7. 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 100
  8. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  9. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  10. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  11. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  12. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  13. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  14. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  15. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  16. 2021Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoodscited by 61
  17. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  18. 2023One-Step Diffusion Distillation via Deep Equilibrium Modelscited by 60
  19. 2021Single-Shot Motion Completion with Transformercited by 59
  20. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  21. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  22. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  23. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  24. 2021Representation learning for neural population activity with Neural Data Transformerscited by 48
  25. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  26. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  27. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  28. 2025HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policycited by 42
  29. 2022Relational Attention: Generalizing Transformers for Graph-Structured Taskscited by 41
  30. 2022Evaluating Long-Term Memory in 3D Mazescited by 41
  31. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  32. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  33. 2022Transformers are Meta-Reinforcement Learnerscited by 39
  34. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  35. 2024Mastering Memory Tasks with World Modelscited by 38
  36. 2021Tell me why! Explanations support learning relational and causal structurecited by 36
  37. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  38. 2021Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agentscited by 35

50 most cited of 262 All 262 (JSON)