# How Crucial is Transformer in Decision Transformer?

arXiv: https://arxiv.org/abs/2211.14655 · HTML: https://mlsanity.com/paper/2211.14655/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions](https://mlsanity.com/paper/1912.02875/) (2019)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks](https://mlsanity.com/paper/1909.09586/) (2019)
- [Can Wikipedia Help Offline Reinforcement Learning?](https://mlsanity.com/paper/2201.12122/) (2022)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks](https://mlsanity.com/paper/2112.02845/) (2021)

## Built on by (most cited first, 14 of 14)

- [A Survey on Transformers in Reinforcement Learning](https://mlsanity.com/paper/2301.03044/) (2023)
- [Generalization to New Sequential Decision Making Tasks with In-Context Learning](https://mlsanity.com/paper/2312.03801/) (2023)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Learning to Modulate pre-trained Models in RL](https://mlsanity.com/paper/2306.14884/) (2023)
- [Critic-Guided Decision Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2312.13716/) (2023)
- [Zero-Shot Reinforcement Learning from Low Quality Data](https://mlsanity.com/paper/2309.15178/) (2023)
- [Recurrent Action Transformer with Memory](https://mlsanity.com/paper/2306.09459/) (2023)
- [Graph World Model](https://mlsanity.com/paper/2507.10539/) (2025)
- [A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks](https://mlsanity.com/paper/2410.22391/) (2024)
- [Representation Matters for Mastering Chess: Improved Feature Representation in AlphaZero Outperforms Switching to Transformers](https://mlsanity.com/paper/2304.14918/) (2023)
- [Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning](https://mlsanity.com/paper/2406.02013/) (2024)
- [Decision SpikeFormer: Spike-Driven Transformer for Decision Making](https://mlsanity.com/paper/2504.03800/) (2025)
- [Accelerating Matrix Diagonalization through Decision Transformers with Epsilon-Greedy Optimization](https://mlsanity.com/paper/2406.16191/) (2024)
- [Self-Supervised Keyframe Discovery for Horizon-Invariant Behavior Cloning](https://mlsanity.com/paper/2610.10857/) (2026)

## Abstract (arXiv)

Decision Transformer (DT) is a recently proposed architecture for Reinforcement Learning that frames the decision-making process as an auto-regressive sequence modeling problem and uses a Transformer model to predict the next action in a sequence of states, actions, and rewards. In this paper, we analyze how crucial the Transformer model is in the complete DT architecture on continuous control tasks. Namely, we replace the Transformer by an LSTM model while keeping the other parts unchanged to obtain what we call a Decision LSTM model. We compare it to DT on continuous control tasks, including pendulum swing-up and stabilization, in simulation and on physical hardware. Our experiments show that DT struggles with continuous control problems, such as inverted pendulum and Furuta pendulum stabilization. On the other hand, the proposed Decision LSTM is able to achieve expert-level performance on these tasks, in addition to learning a swing-up controller on the real system. These results suggest that the strength of the Decision Transformer for continuous control tasks may lie in the overall sequential modeling architecture and not in the Transformer per se.