mlsanityData to 7 Oct 2026

How Crucial is Transformer in Decision Transformer?

arXiv 2211.14655 ↗First posted 26 Nov 2022cs.LG cs.ROcited by 14

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Decision Transformer (DT) is a recently proposed architecture for Reinforcement Learning that frames the decision-making process as an auto-regressive sequence modeling problem and uses a Transformer model to predict the next action in a sequence of states, actions, and rewards. In this paper, we analyze how crucial the Transformer model is in the complete DT architecture on continuous control tasks. Namely, we replace the Transformer by an LSTM model while keeping the other parts unchanged to obtain what we call a Decision LSTM model. We compare it to DT on continuous control tasks, including pendulum swing-up and stabilization, in simulation and on physical hardware. Our experiments show that DT struggles with continuous control problems, such as inverted pendulum and Furuta pendulum stabilization. On the other hand, the proposed Decision LSTM is able to achieve expert-level performance on these tasks, in addition to learning a swing-up controller on the real system. These results suggest that the strength of the Decision Transformer for continuous control tasks may lie in the overall sequential modeling architecture and not in the Transformer per se.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 30 references that are in the map. First posted on arXiv 26 Nov 2022.

Builds on

22 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Online Decision Transformer11 Feb
  2. 2022Can Wikipedia Help Offline Reinforcement Learning?28 Jan
  3. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2016OpenAI Gymcited by 1,684
  8. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2021Transformers in Vision: A Surveycited by 1,009
  11. 2022A Generalist Agentcited by 802
  12. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
5 more
  1. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  2. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  3. 2019Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networkscited by 103
  4. 2021Robot Learning from Randomized Simulations: A Reviewcited by 56
  5. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 3; 2025: 2; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  2. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  3. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  4. 2023Learning to Modulate pre-trained Models in RLcited by 28
  5. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 25
  6. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  7. 2023Recurrent Action Transformer with Memorycited by 17
  8. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  9. 2025Graph World Modelcited by 14
  10. 2023Representation Matters for Mastering Chess: Improved Feature Representation in AlphaZero Outperforms Switching to Transformerscited by 8
  11. 2024Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learningcited by 1
  12. 2025Decision SpikeFormer: Spike-Driven Transformer for Decision Makingcited by 1
2 more
  1. 2024Accelerating Matrix Diagonalization through Decision Transformers with Epsilon-Greedy Optimizationcited by 0
  2. 2026Self-Supervised Keyframe Discovery for Horizon-Invariant Behavior Cloningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  2. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  3. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Control
  4. 2026Decoupling Return-to-Go for Efficient Decision Transformer
  5. 2024Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces
  6. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
  7. 2024Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games
  8. 2025DRDT3: Diffusion-Refined Decision Test-Time Training Model