mlsanityData to 7 Oct 2026

A Survey on Transformers in Reinforcement Learning

arXiv 2301.03044 ↗First posted 8 Jan 2023cs.LG cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

Transformer has been considered the dominating neural architecture in NLP and CV, mostly under supervised settings. Recently, a similar surge of using Transformers has appeared in the domain of reinforcement learning (RL), but it is faced with unique design choices and challenges brought by the nature of RL. However, the evolution of Transformers in RL has not yet been well unraveled. In this paper, we seek to systematically review motivations and progress on using Transformers in RL, provide a taxonomy on existing works, discuss each sub-field, and summarize future prospects.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 111 of its 146 references that are in the map. First posted on arXiv 8 Jan 2023.

Builds on

111 of 146 references.

References · 111

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  11. 2016Densely Connected Convolutional Networkscited by 6,691
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  13. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  14. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  15. 2017Hindsight Experience Replaycited by 1,096
  16. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  17. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  18. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  19. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  20. 2021Transformers in Vision: A Surveycited by 1,009
  21. 2020Mastering Atari with Discrete World Modelscited by 990
  22. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  23. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  24. 2022A Generalist Agentcited by 802
  25. 2020Efficient Transformers: A Surveycited by 784
  26. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  27. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  28. 2017Deep Reinforcement Learning that Matterscited by 727
  29. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  30. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  31. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  32. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  33. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  34. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  35. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  36. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  37. 2022Transformers are Sample-Efficient World Modelscited by 317
  38. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308

50 most cited of 111 All 111 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 15; 2025: 11; 2026: 9.

50 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  3. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  4. 2023Transformers in Speech Processing: A Surveycited by 38
  5. 2023Learning to Modulate pre-trained Models in RLcited by 28
  6. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 25
  7. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 23
  8. 2024Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agentcited by 20
  9. 2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
  10. 2025Softmax is 1/2-Lipschitz: A tight bound across all ℓp normscited by 18
  11. 2023Recurrent Action Transformer with Memorycited by 17
  12. 2025Generative Models in Decision Making: A Surveycited by 14
38 more
  1. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 13
  2. 2024Vision Transformers for End-to-End Vision-Based Quadrotor Obstacle Avoidancecited by 10
  3. 2023Pre-training with Synthetic Data Helps Offline Reinforcement Learningcited by 9
  4. 2023Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actionscited by 8
  5. 2023Representation Matters for Mastering Chess: Improved Feature Representation in AlphaZero Outperforms Switching to Transformerscited by 8
  6. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?cited by 7
  7. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  8. 2024Attention as an RNNcited by 6
  9. 2023PDiT: Interleaving Perception and Decision-making Transformers for Deep Reinforcement Learningcited by 5
  10. 2024Quantum feedback control with a transformer neural network architecturecited by 5
  11. 2023PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Taskscited by 4
  12. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  13. 2023Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approachcited by 3
  14. 2024BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplaycited by 3
  15. 2025Beam Selection in ISAC using Contextual Bandit with Multi-modal Transformer and Transfer Learningcited by 3
  16. 2024Federated Prompt-based Decision Transformer for Customized VR Services in Mobile Edge Computing Systemcited by 2
  17. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  18. 2025Transformers as Unsupervised Learning Algorithms: A study on Gaussian Mixturescited by 2
  19. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?cited by 2
  20. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 2
  21. 2024Attention-Based Deep Reinforcement Learning for Qubit Allocation in Modular Quantum Architecturescited by 1
  22. 2024DODT: Enhanced Online Decision Transformer Learning through Dreamer's Actor-Critic Trajectory Forecastingcited by 1
  23. 2025Transformer-Empowered Actor-Critic Reinforcement Learning for Sequence-Aware Service Function Chain Partitioningcited by 1
  24. 2025Growing with Experience: Growing Neural Networks in Deep Reinforcement Learningcited by 1
  25. 2026In-Context Reinforcement Learning From Suboptimal Historical Datacited by 1
  26. 2023Large-scale Foundation Models and Generative AI for BigData Neurosciencecited by 0
  27. 2024Localized Gaussians as Self-Attention Weights for Point Clouds Correspondencecited by 0
  28. 2024Cooperative and Asynchronous Transformer-based Mission Planning for Heterogeneous Teams of Mobile Robotscited by 0
  29. 2025Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-Task Offline RLcited by 0
  30. 2025Diverse Transformer Decoding for Offline Reinforcement Learning Using Financial Algorithmic Approachescited by 0
  31. 2025Pretraining in Actor-Critic Reinforcement Learning for Locomotioncited by 0
  32. 2025Enhancing Nuclear Reactor Core Simulation through Data-Based Surrogate Modelscited by 0
  33. 2026Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformerscited by 0
  34. 2026Knowledge Distillation for Efficient Transformer-Based Reinforcement Learning in Hardware-Constrained Energy Management Systemscited by 0
  35. 2026CA-AC-MPC: CUDA-Accelerated Actor-Critic Model Predictive Controlcited by 0
  36. 2026Transformer-Enhanced Reinforcement Learning: Fundamentals and Applications in Communication Networkscited by 0
  37. 2026Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Makingcited by 0
  38. 2026Generative Optimization for Incentivized Advertising with Global Level Constraintscited by 0