# When should we prefer Decision Transformers for Offline Reinforcement Learning?

arXiv: https://arxiv.org/abs/2305.14550 · HTML: https://mlsanity.com/paper/2305.14550/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 14 of its 56 references that are in the map.
- First posted on arXiv 23 May 2023.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios](https://mlsanity.com/paper/2212.11419/) (2022)
- [Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions](https://mlsanity.com/paper/1912.02875/) (2019)
- [Training Agents using Upside-Down Reinforcement Learning](https://mlsanity.com/paper/1912.02877/) (2019)
- [Reward-Conditioned Policies](https://mlsanity.com/paper/1912.13465/) (2019)
- [Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.13425/) (2022)
- [You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments](https://mlsanity.com/paper/2205.15967/) (2022)
- [Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL](https://mlsanity.com/paper/2206.00695/) (2022)

## Built on by (most cited first, 20 of 20)

- [Q-value Regularized Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.17098/) (2024)
- [Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning](https://mlsanity.com/paper/2310.20587/) (2023)
- [Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement](https://mlsanity.com/paper/2410.11448/) (2024)
- [Structure in Deep Reinforcement Learning: A Survey and Open Problems](https://mlsanity.com/paper/2306.16021/) (2023)
- [Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learning](https://mlsanity.com/paper/2310.19308/) (2023)
- [Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.23450/) (2024)
- [Efficient Planning with Latent Diffusion](https://mlsanity.com/paper/2310.00311/) (2023)
- [Multi-agent Coordination via Flow Matching](https://mlsanity.com/paper/2511.05005/) (2025)
- [Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning](https://mlsanity.com/paper/2506.12529/) (2025)
- [Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?](https://mlsanity.com/paper/2507.10174/) (2025)
- [Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning](https://mlsanity.com/paper/2406.02013/) (2024)
- [DRDT3: Diffusion-Refined Decision Test-Time Training Model](https://mlsanity.com/paper/2501.06718/) (2025)
- [Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer](https://mlsanity.com/paper/2505.09114/) (2025)
- [Offline Reinforcement Learning for Mobility Robustness Optimization](https://mlsanity.com/paper/2506.22793/) (2025)
- [A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms](https://mlsanity.com/paper/2511.16475/) (2025)
- [Action-Free Offline-to-Online RL via Discretised State Policies](https://mlsanity.com/paper/2602.00629/) (2026)
- [Selecting Offline Reinforcement Learning Algorithms for Stochastic Network Control](https://mlsanity.com/paper/2603.03932/) (2026)
- [Revealing Safety-Critical Scenarios for UTM via Transformer](https://mlsanity.com/paper/2606.31114/) (2026)
- [WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimization](https://mlsanity.com/paper/2609.00284/) (2026)
- [JumpStart Your Policy Learning with Lessons from 160,000 Training Runs](https://mlsanity.com/paper/2609.13730/) (2026)

## Abstract (arXiv)

Offline reinforcement learning (RL) allows agents to learn effective, return-maximizing policies from a static dataset. Three popular algorithms for offline RL are Conservative Q-Learning (CQL), Behavior Cloning (BC), and Decision Transformer (DT), from the class of Q-Learning, Imitation Learning, and Sequence Modeling respectively. A key open question is: which algorithm is preferred under what conditions? We study this question empirically by exploring the performance of these algorithms across the commonly used D4RL and Robomimic benchmarks. We design targeted experiments to understand their behavior concerning data suboptimality, task complexity, and stochasticity. Our key findings are: (1) DT requires more data than CQL to learn competitive policies but is more robust; (2) DT is a substantially better choice than both CQL and BC in sparse-reward and low-quality data settings; (3) DT and BC are preferable as task horizon increases, or when data is obtained from human demonstrators; and (4) CQL excels in situations characterized by the combination of high stochasticity and low data quality. We also investigate architectural choices and scaling trends for DT on Atari and D4RL and make design/scaling recommendations. We find that scaling the amount of data for DT by 5x gives a 2.5x average score improvement on Atari.