mlsanityData to 14 Oct 2026

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning

arXiv 2506.07744 ↗First posted 9 Jun 2025cs.LG cs.AI cs.ROcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Existing offline hierarchical reinforcement learning methods rely on high-level policy learning to generate subgoal sequences. However, their efficiency degrades as task horizons increase, and they lack effective strategies for stitching useful state transitions across different trajectories. We propose Graph-Assisted Stitching (GAS), a novel framework that formulates subgoal selection as a graph search problem rather than learning an explicit high-level policy. By embedding states into a Temporal Distance Representation (TDR) space, GAS clusters semantically similar states from different trajectories into unified graph nodes, enabling efficient transition stitching. A shortest-path algorithm is then applied to select subgoal sequences within the graph, while a low-level policy learns to reach the subgoals. To improve graph quality, we introduce the Temporal Efficiency (TE) metric, which filters out noisy or inefficient transition states, significantly enhancing task performance. GAS outperforms prior offline HRL methods across locomotion, navigation, and manipulation tasks. Notably, in the most stitching-critical task, it achieves a score of 88.3, dramatically surpassing the previous state-of-the-art score of 1.0. Our source code is available at: https://github.com/qortmdgh4141/GAS.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 65 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

46 of 65 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
  2. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching4 Feb
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  6. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 37

  1. 2016Layer Normalizationcited by 5,669
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  9. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  11. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  12. 2019Learning Latent Plans from Playcited by 316
25 more
  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  3. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  4. 2025Flow Q-Learningcited by 184
  5. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  6. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  7. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  8. 2024Foundation Policies with Hilbert Representationscited by 94
  9. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 89
  10. 2017PRM-RL: Long-range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planningcited by 69
  11. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
  12. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
  13. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  14. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
  15. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  16. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 27
  17. 2021Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learningcited by 23
  18. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 23
  19. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  20. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RLcited by 21
  21. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  22. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 19
  23. 2023Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skillscited by 14
  24. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
  25. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learningcited by 8
  2. 2026SVL: Goal-Conditioned Reinforcement Learning as Survival Learningcited by 3
  3. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
  4. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learningcited by 2
  5. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 2
  6. 2026Hitting Time Isomorphism for Multi-Stage Planning with Foundation Policiescited by 1
  7. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
  8. 2026GraSP-STL: A Graph-Based Framework for Zero-Shot Signal Temporal Logic Planning via Offline Goal-Conditioned Reinforcement Learningcited by 0
  9. 2026Plan First, Diffuse Later: Extrinsic Graph Guidance for Long-Horizon Diffusion Planningcited by 0
  10. 2026ChronoForest: Closed-Loop Multi-Tree Diffusion Planning for Efficient Bridge Search and Route Compositioncited by 0
  11. 2026Metro-WM: Long-Horizon Latent Planning with Realisable Sub-Goalscited by 0
  12. 2026Learning Goal-Reaching Quasimetric Geometry From Finite-Time Reachabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learning
  2. 2025ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rollouts
  3. 2025State-Covering Trajectory Stitching for Diffusion Planners
  4. 2024Context-Former: Stitching via Latent Conditioned Sequence Modeling
  5. 2025Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representations
  6. 2026GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RL
  7. 2024Temporal Abstraction in Reinforcement Learning with Offline Data
  8. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitching