mlsanityData to 7 Oct 2026

DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching

arXiv 2402.02439 ↗First posted 4 Feb 2024cs.LG cs.AIcited by 35

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning (RL), the performance of the learned policy highly depends on the quality of offline datasets. However, in many cases, the offline dataset contains very limited optimal trajectories, which poses a challenge for offline RL algorithms as agents must acquire the ability to transit to high-reward regions. To address this issue, we introduce Diffusion-based Trajectory Stitching (DiffStitch), a novel diffusion-based data augmentation pipeline that systematically generates stitching transitions between trajectories. DiffStitch effectively connects low-reward trajectories with high-reward trajectories, forming globally optimal trajectories to address the challenges faced by offline RL algorithms. Empirical experiments conducted on D4RL datasets demonstrate the effectiveness of DiffStitch across RL methodologies. Notably, DiffStitch demonstrates substantial enhancements in the performance of one-step methods (IQL), imitation learning methods (TD3+BC), and trajectory optimization methods (DT).

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 31 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

27 of 31 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning10 Apr
  2. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  3. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination16 Jun
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  7. 2021Offline RL Without Off-Policy Evaluation16 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2020Critic Regularized Regression26 Jun
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
1 more
  1. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  7. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  8. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  9. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  10. 2023Synthetic Experience Replaycited by 100
  11. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  12. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
2 more
  1. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 28
  2. 2022Semi-Markov Offline Reinforcement Learning for Healthcarecited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 24; 2026: 6.

35 papers.

Sort:

By area:

  1. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  2. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  3. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
  4. 2025Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planningcited by 8
  5. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  6. 2025Zero-Shot Trajectory Planning for Signal Temporal Logic Taskscited by 6
  7. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learningcited by 4
  8. 2025BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learningcited by 4
  9. 2024Return-Aligned Decision Transformercited by 3
  10. 2025Is Temporal Difference Learning the Gold Standard for Stitching in RL?cited by 3
  11. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policycited by 3
  12. 2024Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generationcited by 2
23 more
  1. 2025Offline Reinforcement Learning via Inverse Optimizationcited by 2
  2. 2025STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluationcited by 2
  3. 2025Compositional Monte Carlo Tree Diffusion for Extendable Planningcited by 2
  4. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  5. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
  6. 2024Offline Trajectory Optimization for Offline Reinforcement Learningcited by 1
  7. 2025DRDT3: Diffusion-Refined Decision Test-Time Training Modelcited by 1
  8. 2025Extendable Planning via Multiscale Diffusioncited by 1
  9. 2025Treatment Stitching with Schr"odinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment Strategiescited by 1
  10. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  11. 2024From Incomplete Coarse-Grained to Complete Fine-Grained: A Two-Stage Framework for Spatiotemporal Data Reconstructioncited by 0
  12. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
  13. 2025Improving Trajectory Stitching with Flow Modelscited by 0
  14. 2025What Do You Need for Compositional Generalization in Diffusion Planning?cited by 0
  15. 2025Decision Flow Policy Optimizationcited by 0
  16. 2025RAD: Retrieval High-quality Demonstrations to Enhance Decision-makingcited by 0
  17. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learningcited by 0
  18. 2025Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shiftcited by 0
  19. 2025ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rolloutscited by 0
  20. 2025SD2AIL: Adversarial Imitation Learning from Synthetic Demonstrations via Diffusion Modelscited by 0
  21. 2026Q-Regularized Generative Auto-Bidding: From Suboptimal Trajectories to Optimal Policiescited by 0
  22. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  23. 2026Trajectory-Level Data Augmentation for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learning
  2. 2023Efficient Diffusion Policies for Offline Reinforcement Learning
  3. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
  4. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  5. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  6. 2024Augmenting Offline Reinforcement Learning with State-only Interactions
  7. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning
  8. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RL