mlsanityData to 13 Oct 2026

Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generation

arXiv 2411.11327 ↗First posted 18 Nov 2024cs.LGcited by 2

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 2 more
Abstract (arXiv)

Decision Transformer (DT) can learn effective policy from offline datasets by converting the offline reinforcement learning (RL) into a supervised sequence modeling task, where the trajectory elements are generated auto-regressively conditioned on the return-to-go (RTG).However, the sequence modeling learning approach tends to learn policies that converge on the sub-optimal trajectories within the dataset, for lack of bridging data to move to better trajectories, even if the condition is set to the highest RTG.To address this issue, we introduce Diffusion-Based Trajectory Branch Generation (BG), which expands the trajectories of the dataset with branches generated by a diffusion model.The trajectory branch is generated based on the segment of the trajectory within the dataset, and leads to trajectories with higher returns.We concatenate the generated branch with the trajectory segment as an expansion of the trajectory.After expanding, DT has more opportunities to learn policies to move to better trajectories, preventing it from converging to the sub-optimal trajectories.Empirically, after processing with BG, DT outperforms state-of-the-art sequence modeling methods on D4RL benchmark, demonstrating the effectiveness of adding branches to the dataset without further modifications.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 24 of its 33 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

24 of 33 references.

References · 24

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,028
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,023
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,889
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,782
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,531
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,415
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,257
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,231
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 884
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 657
12 more
  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 637
  2. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 292
  3. 2020Critic Regularized Regressioncited by 270
  4. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 214
  5. 2021Offline RL Without Off-Policy Evaluationcited by 150
  6. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  7. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  8. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  9. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 36
  10. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  11. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
  12. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 40
  2. 2025Extremum Flow Matching for Offline Goal Conditioned Reinforcement Learningcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025DRDT3: Diffusion-Refined Decision Test-Time Training Model
  2. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  3. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RL
  4. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets
  5. 2026Decoupling Return-to-Go for Efficient Decision Transformer
  6. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer
  7. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning
  8. 2024In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning