# Transformers in Reinforcement Learning: A Survey

arXiv: https://arxiv.org/abs/2307.05979 · HTML: https://mlsanity.com/paper/2307.05979/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 144 of its 220 references that are in the map.
- First posted on arXiv 12 Jul 2023.

## Builds on (most cited first, 30 of 144)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)

## Built on by (most cited first, 12 of 12)

- [Emergence of In-Context Reinforcement Learning from Noise Distillation](https://mlsanity.com/paper/2312.12275/) (2023)
- [Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns](https://mlsanity.com/paper/2503.03660/) (2025)
- [XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning](https://mlsanity.com/paper/2406.08973/) (2024)
- [Attention as an RNN](https://mlsanity.com/paper/2405.13956/) (2024)
- [Retentive Decision Transformer with Adaptive Masking for Reinforcement Learning based Recommendation Systems](https://mlsanity.com/paper/2403.17634/) (2024)
- [Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment](https://mlsanity.com/paper/2407.19380/) (2024)
- [A Survey for Deep Reinforcement Learning Based Network Intrusion Detection](https://mlsanity.com/paper/2410.07612/) (2024)
- [Transformer-Empowered Actor-Critic Reinforcement Learning for Sequence-Aware Service Function Chain Partitioning](https://mlsanity.com/paper/2504.18902/) (2025)
- [TransAM: Transformer-Based Agent Modeling for Multi-Agent Systems via Local Trajectory Encoding](https://mlsanity.com/paper/2508.02826/) (2025)
- [Meta-Adaptive Beam Search Planning for Transformer-Based Reinforcement Learning Control of UAVs with Overhead Manipulators under Flight Disturbances](https://mlsanity.com/paper/2603.26612/) (2026)
- [Transformer-Enhanced Reinforcement Learning: Fundamentals and Applications in Communication Networks](https://mlsanity.com/paper/2606.05208/) (2026)
- [Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models](https://mlsanity.com/paper/2606.24962/) (2026)

## Abstract (arXiv)

Transformers have significantly impacted domains like natural language processing, computer vision, and robotics, where they improve performance compared to other neural networks. This survey explores how transformers are used in reinforcement learning (RL), where they are seen as a promising solution for addressing challenges such as unstable training, credit assignment, lack of interpretability, and partial observability. We begin by providing a brief domain overview of RL, followed by a discussion on the challenges of classical RL algorithms. Next, we delve into the properties of the transformer and its variants and discuss the characteristics that make them well-suited to address the challenges inherent in RL. We examine the application of transformers to various aspects of RL, including representation learning, transition and reward function modeling, and policy optimization. We also discuss recent research that aims to enhance the interpretability and efficiency of transformers in RL, using visualization techniques and efficient training strategies. Often, the transformer architecture must be tailored to the specific needs of a given application. We present a broad overview of how transformers have been adapted for several applications, including robotics, medicine, language modeling, cloud computing, and combinatorial optimization. We conclude by discussing the limitations of using transformers in RL and assess their potential for catalyzing future breakthroughs in this field.