mlsanityData to 7 Oct 2026

Online Decision Transformer

arXiv 2202.05607 ↗First posted 11 Feb 2022cs.LG cs.AIcited by 194

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling. However, any practical instantiation of RL also involves an online component, where policies pretrained on passive offline datasets are finetuned via taskspecific interactions with the environment. We propose Online Decision Transformers (ODT), an RL algorithm based on sequence modeling that blends offline pretraining with online finetuning in a unified framework. Our framework uses sequence-level entropy regularizers in conjunction with autoregressive modeling objectives for sample-efficient exploration and finetuning. Empirically, we show that ODT is competitive with the state-of-the-art in absolute performance on the D4RL benchmark but shows much more significant gains during the finetuning procedure.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 67 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

44 of 67 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  2. 2021Transfer learning with causal counterfactual reasoning in Decision Transformers27 Oct
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2021Convex optimizationcited by 2,479
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  12. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
28 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2019Model-Based Reinforcement Learning for Ataricited by 495
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  13. 2020Performative Predictioncited by 318
  14. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  15. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  16. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  17. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  18. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  19. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  20. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  21. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  22. 2020Stochastic Optimization for Performative Predictioncited by 92
  23. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
  24. 2020Stochastic optimization with decision-dependent distributionscited by 65
  25. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 61
  26. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 43
  27. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  28. 2017Hindsight policy gradientscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 31; 2023: 60; 2024: 58; 2025: 23; 2026: 22.

194 papers.

Sort:

By area:

  1. 2022A Generalist Agentcited by 802
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  5. 2022Transformers are Sample-Efficient World Modelscited by 317
  6. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  7. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  8. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  9. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  10. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  11. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  12. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
38 more
  1. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  2. 2022Jump-Start Reinforcement Learningcited by 102
  3. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  4. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  5. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  6. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  7. 2023Diffusion Models for Black-Box Optimizationcited by 75
  8. 2023Elastic Decision Transformercited by 66
  9. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  10. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  11. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  12. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 59
  13. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  14. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  15. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  16. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  17. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  18. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  19. 2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 44
  20. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  21. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  22. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  23. 2023MTD-GPT: A Multi-Task Decision-Making GPT Model for Autonomous Driving at Unsignalized Intersectionscited by 38
  24. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  25. 2022Generative Pretraining for Black-Box Optimizationcited by 36
  26. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  27. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  28. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
  29. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  30. 2023Hyper-Decision Transformer for Efficient Online Policy Adaptationcited by 32
  31. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  32. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  33. 2022Reward Gaming in Conditional Text Generationcited by 29
  34. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  35. 2023Learning to Modulate pre-trained Models in RLcited by 28
  36. 2022Challenges and Opportunities in Deep Reinforcement Learning with Graph Neural Networks: A Comprehensive review of Algorithms and Applicationscited by 27
  37. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 27
  38. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 27

50 most cited of 194 All 194 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms
  2. 2024Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning
  3. 2026Active Offline-to-Online Reinforcement Learning
  4. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL
  5. 2024Solving Continual Offline Reinforcement Learning with Decision Transformer
  6. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?
  7. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  8. 2022Bootstrapped Transformer for Offline Reinforcement Learning