mlsanityData to 7 Oct 2026

When should we prefer Decision Transformers for Offline Reinforcement Learning?

arXiv 2305.14550 ↗First posted 23 May 2023cs.LG cs.AIcited by 20

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Offline reinforcement learning (RL) allows agents to learn effective, return-maximizing policies from a static dataset. Three popular algorithms for offline RL are Conservative Q-Learning (CQL), Behavior Cloning (BC), and Decision Transformer (DT), from the class of Q-Learning, Imitation Learning, and Sequence Modeling respectively. A key open question is: which algorithm is preferred under what conditions? We study this question empirically by exploring the performance of these algorithms across the commonly used D4RL and Robomimic benchmarks. We design targeted experiments to understand their behavior concerning data suboptimality, task complexity, and stochasticity. Our key findings are: (1) DT requires more data than CQL to learn competitive policies but is more robust; (2) DT is a substantially better choice than both CQL and BC in sparse-reward and low-quality data settings; (3) DT and BC are preferable as task horizon increases, or when data is obtained from human demonstrators; and (4) CQL excels in situations characterized by the combination of high stochasticity and low data quality. We also investigate architectural choices and scaling trends for DT on Atari and D4RL and make design/scaling recommendations. We find that scaling the amount of data for DT by 5x gives a 2.5x average score improvement on Atari.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 56 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

14 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments31 May

Other references · 13

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  7. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  8. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  9. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
  10. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  11. 2019Reward-Conditioned Policiescited by 101
  12. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
1 more
  1. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 7; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024Q-value Regularized Transformer for Offline Reinforcement Learningcited by 28
  2. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  3. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  4. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
  5. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  6. 2023Efficient Planning with Latent Diffusioncited by 11
  7. 2023Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learningcited by 11
  8. 2025Multi-agent Coordination via Flow Matchingcited by 9
  9. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  10. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?cited by 2
  11. 2024Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learningcited by 1
  12. 2025DRDT3: Diffusion-Refined Decision Test-Time Training Modelcited by 1
8 more
  1. 2025Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformercited by 1
  2. 2025Offline Reinforcement Learning for Mobility Robustness Optimizationcited by 1
  3. 2025A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithmscited by 0
  4. 2026Action-Free Offline-to-Online RL via Discretised State Policiescited by 0
  5. 2026Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Controlcited by 0
  6. 2026Revealing Safety-Critical Scenarios for UTM via Transformercited by 0
  7. 2026WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimizationcited by 0
  8. 2026JumpStart Your Policy Learning with Lessons from 160,000 Training Runscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?
  2. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL
  3. 2020Offline Reinforcement Learning Hands-On
  4. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  6. 2020Hyperparameter Selection for Offline Reinforcement Learning
  7. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  8. 2024Solving Continual Offline Reinforcement Learning with Decision Transformer