mlsanityData to 14 Oct 2026

Model-based Trajectory Stitching for Improved Offline Reinforcement Learning

arXiv 2211.11603 ↗First posted 21 Nov 2022cs.LG stat.MLcited by 15

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In many real-world applications, collecting large and high-quality datasets may be too costly or impractical. Offline reinforcement learning (RL) aims to infer an optimal decision-making policy from a fixed set of data. Getting the most information from historical data is then vital for good performance once the policy is deployed. We propose a model-based data augmentation strategy, Trajectory Stitching (TS), to improve the quality of sub-optimal historical trajectories. TS introduces unseen actions joining previously disconnected states: using a probabilistic notion of state reachability, it effectively `stitches' together parts of the historical demonstrations to generate new, higher quality ones. A stitching event consists of a transition between a pair of observed states through a synthetic and highly probable action. New actions are introduced only when they are expected to be beneficial, according to an estimated state-value function. We show that using this data augmentation strategy jointly with behavioural cloning (BC) leads to improvements over the behaviour-cloned policy from the original dataset. Improving over the BC policy could then be used as a launchpad for online RL through planning and demonstration-guided RL.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 67 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

46 of 67 references.

Closest ancestors · 17

Same direction, newest first

  1. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  2. 2022BATS: Best Action Trajectory Stitching26 Apr
  3. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  8. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  9. 2021Offline Reinforcement Learning with Pseudometric Learning2 Mar
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
5 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2013Auto-Encoding Variational Bayescited by 4,053
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
17 more
  1. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  2. 2021Implicit Behavioral Cloningcited by 452
  3. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  4. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  5. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  6. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 346
  7. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  9. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  10. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  11. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  12. 2020Model-Based Offline Planningcited by 129
  13. 2019Imitation Learning as f-Divergence Minimizationcited by 123
  14. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  15. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  16. 2021Counter-Strike Deathmatch with Large-Scale Behavioural Cloningcited by 50
  17. 2021Towards Robust Bisimulation Metric Learningcited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 4; 2025: 4; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023Elastic Decision Transformercited by 67
  2. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 41
  3. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 27
  4. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  5. 2023Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learningcited by 13
  6. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10
  7. 2022Model-based trajectory stitching for improved behavioural cloning and its applicationscited by 9
  8. 2023Budgeting Counterfactual for Offline RLcited by 4
  9. 2025Is Temporal Difference Learning the Gold Standard for Stitching in RL?cited by 4
  10. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policycited by 4
  11. 2024Decision-Point Guided Safe Policy Improvementcited by 2
  12. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
3 more
  1. 2024State-Constrained Offline Reinforcement Learningcited by 1
  2. 2025Improving Trajectory Stitching with Flow Modelscited by 0
  3. 2025What Do You Need for Compositional Generalization in Diffusion Planning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching
  2. 2025ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rollouts
  3. 2023Offline RL with Observation Histories: Analyzing and Improving Sample Complexity
  4. 2026Trajectory-Level Data Augmentation for Offline Reinforcement Learning
  5. 2025State-Covering Trajectory Stitching for Diffusion Planners
  6. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
  7. 2024Augmenting Offline Reinforcement Learning with State-only Interactions
  8. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting