# Q-value Regularized Transformer for Offline Reinforcement Learning arXiv: https://arxiv.org/abs/2405.17098 ยท HTML: https://mlsanity.com/paper/2405.17098/ ## Key facts - Cited by 29 papers in the map (data through 14 Oct 2026). - Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. - Builds on 22 of its 53 references that are in the map. - First posted on arXiv 27 May 2024. ## Placed in - Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/) ## Builds on (most cited first, 22 of 22) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020) - [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020) - [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021) - [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022) - [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019) - [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019) - [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022) - [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022) - [Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning](https://mlsanity.com/paper/2002.08396/) (2020) - [RvS: What is Essential for Offline RL via Supervised Learning?](https://mlsanity.com/paper/2112.10751/) (2021) - [Training Agents using Upside-Down Reinforcement Learning](https://mlsanity.com/paper/1912.02877/) (2019) - [Reward-Conditioned Policies](https://mlsanity.com/paper/1912.13465/) (2019) - [Elastic Decision Transformer](https://mlsanity.com/paper/2307.02484/) (2023) - [Dichotomy of Control: Separating What You Can Control from What You Cannot](https://mlsanity.com/paper/2210.13435/) (2022) - [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022) - [Critic-Guided Decision Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2312.13716/) (2023) - [Graph Decision Transformer](https://mlsanity.com/paper/2303.03747/) (2023) - [When should we prefer Decision Transformers for Offline Reinforcement Learning?](https://mlsanity.com/paper/2305.14550/) (2023) - [Prompt-Tuning Decision Transformer with Preference Ranking](https://mlsanity.com/paper/2305.09648/) (2023) ## Built on by (most cited first, 29 of 29) - [HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.18080/) (2024) - [Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.23450/) (2024) - [Yes, Q-learning Helps Offline In-Context RL](https://mlsanity.com/paper/2502.17666/) (2025) - [Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2407.04285/) (2024) - [Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization](https://mlsanity.com/paper/2411.01168/) (2024) - [Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning](https://mlsanity.com/paper/2409.08062/) (2024) - [Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning](https://mlsanity.com/paper/2505.01822/) (2025) - [Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?](https://mlsanity.com/paper/2507.10174/) (2025) - [Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2511.02567/) (2025) - [Task-Aware Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2411.01146/) (2024) - [Continual Task Learning through Adaptive Policy Self-Composition](https://mlsanity.com/paper/2411.11364/) (2024) - [Decision SpikeFormer: Spike-Driven Transformer for Decision Making](https://mlsanity.com/paper/2504.03800/) (2025) - [How to Provably Improve Return Conditioned Supervised Learning?](https://mlsanity.com/paper/2506.08463/) (2025) - [Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning](https://mlsanity.com/paper/2602.07441/) (2026) - [Communication Learning in Multi-Agent Systems from Graph Modeling Perspective](https://mlsanity.com/paper/2411.00382/) (2024) - [Consensus Based Stochastic Control](https://mlsanity.com/paper/2501.17801/) (2025) - [GNN-DT: Graph Neural Network Enhanced Decision Transformer for Efficient Optimization in Dynamic Environments](https://mlsanity.com/paper/2502.01778/) (2025) - [Mental Accounts for Actions: EWA-Inspired Attention in Decision Transformers](https://mlsanity.com/paper/2509.15498/) (2025) - [In-Context Curiosity: Distilling Exploration for Decision-Pretrained Transformers on Bandit Tasks](https://mlsanity.com/paper/2510.00347/) (2025) - [Adjusting the Output of Decision Transformer with Action Gradient](https://mlsanity.com/paper/2510.05285/) (2025) - [Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shift](https://mlsanity.com/paper/2511.09173/) (2025) - [Conditional Sequence Modeling for Safe Reinforcement Learning](https://mlsanity.com/paper/2602.08584/) (2026) - [CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learning](https://mlsanity.com/paper/2602.09207/) (2026) - [Return-to-Go Is More Than a Number: Q-Guided Alignment for Return-Conditioned Supervised Learning](https://mlsanity.com/paper/2605.29028/) (2026) - [Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning](https://mlsanity.com/paper/2608.02332/) (2026) - [Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management](https://mlsanity.com/paper/2608.28878/) (2026) - [From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning](https://mlsanity.com/paper/2609.30391/) (2026) - [Q-learning Penalized Transformer for Safe Offline Reinforcement Learning](https://mlsanity.com/paper/2609.34426/) (2026) - [Alignment-Guided Flow Transformer for Efficient Vision-Language-Action Policy Learning](https://mlsanity.com/paper/2609.34467/) (2026) ## Abstract (arXiv) Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Conditional Sequence Modeling (CSM), a paradigm that learns the action distribution based on history trajectory and target returns for each state. However, these methods often struggle with stitching together optimal trajectories from sub-optimal ones due to the inconsistency between the sampled returns within individual trajectories and the optimal returns across multiple trajectories. Fortunately, Dynamic Programming (DP) methods offer a solution by leveraging a value function to approximate optimal future returns for each state, while these techniques are prone to unstable learning behaviors, particularly in long-horizon and sparse-reward scenarios. Building upon these insights, we propose the Q-value regularized Transformer (QT), which combines the trajectory modeling ability of the Transformer with the predictability of optimal future returns from DP methods. QT learns an action-value function and integrates a term maximizing action-values into the training loss of CSM, which aims to seek optimal actions that align closely with the behavior policy. Empirical evaluations on D4RL benchmark datasets demonstrate the superiority of QT over traditional DP and CSM methods, highlighting the potential of QT to enhance the state-of-the-art in offline RL.