mlsanityData to 7 Oct 2026

Decision Transformer: Reinforcement Learning via Sequence Modeling

arXiv 2106.01345 ↗First posted 2 Jun 2021cs.LG cs.AIcited by 1,528

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Key facts. Cited by 1,528 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 91 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

56 of 91 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2016Layer Normalizationcited by 5,579
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
38 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2020Mastering Atari with Discrete World Modelscited by 990
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  6. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  13. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  14. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  15. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  16. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  17. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  18. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  19. 2017Toward Controlled Generation of Textcited by 271
  20. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  21. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  22. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  23. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  24. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  25. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  26. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  27. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  28. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  29. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  30. 2018Learning to Write with Cooperative Discriminatorscited by 119
  31. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  32. 2019Reward-Conditioned Policiescited by 101
  33. 2020Transformers for One-Shot Visual Imitationcited by 77
  34. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  35. 2019Hindsight Credit Assignmentcited by 74
  36. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 60
  37. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 57
  38. 2020Imitating Interactive Intelligencecited by 55

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 37; 2022: 207; 2023: 336; 2024: 398; 2025: 298; 2026: 252.

1,528 papers.

Sort:

By area:

  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  7. 2022A Generalist Agentcited by 802
  8. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  9. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
38 more
  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  3. 2022Transformers in Time Series: A Surveycited by 489
  4. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  5. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  6. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  7. 2024Understanding the planning of LLM agents: A surveycited by 414
  8. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  9. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 385
  10. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  11. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  12. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  13. 2022Transformers are Sample-Efficient World Modelscited by 317
  14. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  15. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  16. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  17. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  18. 2026Reinforcement Learning via Self-Distillationcited by 285
  19. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  20. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  21. 2022Transformers Learn Shortcuts to Automatacited by 260
  22. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  23. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  24. 2023Pretraining Language Models with Human Preferencescited by 245
  25. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  26. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  27. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  28. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  29. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  30. 2022Multi-Game Decision Transformerscited by 212
  31. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
  32. 2022TALM: Tool Augmented Language Modelscited by 197
  33. 2023Large Language Models as General Pattern Machinescited by 196
  34. 2022Online Decision Transformercited by 194
  35. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  36. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  37. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  38. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177

50 most cited of 1,528 All 1,528 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Q-value Regularized Transformer for Offline Reinforcement Learning
  2. 2024Learning to Play Atari in a World of Tokens
  3. 2020Learning to Generalize for Sequential Decision Making
  4. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner
  5. 2025Reinforcement Learning for Large Model: A Survey
  6. 2026Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers
  7. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  8. 2026ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policies