mlsanityData to 7 Oct 2026

Offline Reinforcement Learning as One Big Sequence Modeling Problem

arXiv 2106.02039 ↗First posted 3 Jun 2021cs.LG cs.AIcited by 636

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) is typically concerned with estimating stationary policies or single-step models, leveraging the Markov property to factorize problems in time. However, we can also view RL as a generic sequence modeling problem, with the goal being to produce a sequence of actions that leads to a sequence of high rewards. Viewed in this way, it is tempting to consider whether high-capacity sequence prediction models that work well in other domains, such as natural-language processing, can also provide effective solutions to the RL problem. To this end, we explore how RL can be tackled with the tools of sequence modeling, using a Transformer architecture to model distributions over trajectories and repurposing beam search as a planning algorithm. Framing RL as sequence modeling problem simplifies a range of design decisions, allowing us to dispense with many of the components common in offline RL algorithms. We demonstrate the flexibility of this approach across long-horizon dynamics prediction, imitation learning, goal-conditioned RL, and offline RL. Further, we show that this approach can be combined with existing model-free algorithms to yield a state-of-the-art planner in sparse-reward, long-horizon tasks.

Key facts. Cited by 636 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 50 of its 71 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

50 of 71 references.

Closest ancestors · 5

Same direction, newest first

  1. 2019Exploring Model-based Planning with Policy Networks20 Jun
  2. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  4. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb
  5. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning8 Aug

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
33 more
  1. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  2. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  7. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  8. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  9. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  10. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  11. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  12. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  13. 2020Model-Based Offline Planningcited by 126
  14. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  15. 2020If beam search is the answer, what was the question?cited by 114
  16. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
  17. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  18. 2019Reward-Conditioned Policiescited by 101
  19. 2015Memory-based control with recurrent neural networkscited by 77
  20. 2017Recurrent Environment Simulatorscited by 68
  21. 2016Control of Memory, Active Perception, and Action in Minecraftcited by 61
  22. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
  23. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  24. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 46
  25. 2019Insertion-based Decoding with automatically Inferred Generation Ordercited by 43
  26. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 43
  27. 2017Hindsight policy gradientscited by 36
  28. 2020Planning from Pixels using Inverse Dynamics Modelscited by 36
  29. 2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
  30. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  31. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  32. 2021Offline Reinforcement Learning with Pseudometric Learningcited by 28
  33. 2020Adaptive Transformers in RLcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 94; 2023: 165; 2024: 176; 2025: 111; 2026: 79.

636 papers.

Sort:

By area:

  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  3. 2022A Generalist Agentcited by 802
  4. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  5. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  7. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  8. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  9. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  11. 2022Transformers are Sample-Efficient World Modelscited by 317
  12. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
38 more
  1. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  2. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  5. 2022Transformers Learn Shortcuts to Automatacited by 260
  6. 2023Pretraining Language Models with Human Preferencescited by 245
  7. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  8. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  9. 2022Multi-Game Decision Transformerscited by 212
  10. 2022Online Decision Transformercited by 194
  11. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  12. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  13. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  14. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  15. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  16. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  17. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  18. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  19. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  20. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  21. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  22. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  23. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  24. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  25. 2023Large Language Models for Robotics: A Surveycited by 132
  26. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  27. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  28. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  29. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  30. 2023In-Context Convergence of Transformerscited by 112
  31. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  32. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  33. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  34. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  35. 2022Jump-Start Reinforcement Learningcited by 102
  36. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  37. 2022Vision Transformers provably learn spatial structurecited by 92
  38. 2023Compositional Foundation Models for Hierarchical Planningcited by 92

50 most cited of 636 All 636 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  2. 2022Can Wikipedia Help Offline Reinforcement Learning?
  3. 2024Reinformer: Max-Return Sequence Modeling for Offline RL
  4. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL
  5. 2024Are Expressive Models Truly Necessary for Offline RL?
  6. 2024Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning
  7. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  8. 2025Scalable Offline Model-Based RL with Action Chunks