Data to 7 Oct 2026

Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL

arXiv 2209.03993 ↗First posted 8 Sep 2022cs.LGcited by 103

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Recent works have shown that tackling offline reinforcement learning (RL) with a conditional policy produces promising results. The Decision Transformer (DT) combines the conditional policy approach and a transformer architecture, showing competitive performance against several benchmarks. However, DT lacks stitching ability -- one of the critical abilities for offline RL to learn the optimal policy from sub-optimal trajectories. This issue becomes particularly significant when the offline dataset only contains sub-optimal trajectories. On the other hand, the conventional RL approaches based on Dynamic Programming (such as Q-learning) do not have the same limitation; however, they suffer from unstable learning behaviours, especially when they rely on function approximation in an off-policy learning setting. In this paper, we propose the Q-learning Decision Transformer (QDT) to address the shortcomings of DT by leveraging the benefits of Dynamic Programming (Q-learning). It utilises the Dynamic Programming results to relabel the return-to-go in the training data to then train the DT with the relabelled data. Our approach efficiently exploits the benefits of these two approaches and compensates for each other's shortcomings to achieve better performance. We empirically show these in both simple toy environments and the more complex D4RL benchmark, showing competitive performance gains.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 52 references that are in the map. First posted on arXiv 8 Sep 2022.

Builds on

36 of 52 references.

Closest ancestors · 2

Same direction, newest first

Other references · 34

2017Attention Is All You Needcited by 59,932
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2021Zero-Shot Text-to-Image Generationcited by 4,276
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2019Behavior Regularized Offline Reinforcement Learningcited by 654

22 more

2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
2017End-to-end Driving via Conditional Imitation Learningcited by 479
2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
2019Learning Latent Plans from Playcited by 310
2020Critic Regularized Regressioncited by 269
2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
2019Training Agents using Upside-Down Reinforcement Learningcited by 106
2019Reward-Conditioned Policiescited by 101
2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
2021Benchmarks for Deep Off-Policy Evaluationcited by 89
2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
2019Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Targetcited by 31
2013Off-policy Learning with Eligibility Traces: A Surveycited by 12
2021Quantile Filtered Imitation Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 19; 2024: 35; 2025: 26; 2026: 22.

103 papers.

Sort:

By area:

2025π*0.6: a VLA That Learns From Experiencecited by 280
2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
2023Elastic Decision Transformercited by 66
2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
2023A Survey on Transformers in Reinforcement Learningcited by 50
2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 26

38 more

2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 25
2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2023Graph Decision Transformercited by 20
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 18
2023Recurrent Action Transformer with Memorycited by 17
2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
2024Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inferencecited by 14
2024HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learningcited by 14
2023Transformers in Reinforcement Learning: A Surveycited by 11
2023Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learningcited by 11
2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
2024Rethinking Transformers in Solving POMDPscited by 9
2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
2025Yes, Q-learning Helps Offline In-Context RLcited by 8
2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
2024PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performercited by 6
2024Predictive Coding for Decision Transformercited by 6
2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
2025Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learningcited by 6
2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
2023PASTA: Pretrained Action-State Transformer Agentscited by 5
2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
2024GARLIC: GPT-Augmented Reinforcement Learning with Intelligent Control for Vehicle Dispatchingcited by 5
2025Scalable In-Context Q-Learningcited by 5
2024Safe and Robust Reinforcement Learning: Principles and Practicecited by 4

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way