mlsanityData to 14 Oct 2026

Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning

arXiv 2205.11790 ↗First posted 24 May 2022cs.RO cs.SY eess.SYcited by 50

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline Reinforcement learning (RL) has shown potent in many safe-critical tasks in robotics where exploration is risky and expensive. However, it still struggles to acquire skills in temporally extended tasks. In this paper, we study the problem of offline RL for temporally extended tasks. We propose a hierarchical planning framework, consisting of a low-level goal-conditioned RL policy and a high-level goal planner. The low-level policy is trained via offline RL. We improve the offline training to deal with out-of-distribution goals by a perturbed goal sampling process. The high-level planner selects intermediate sub-goals by taking advantages of model-based planning methods. It plans over future sub-goal sequences based on the learned value function of the low-level policy. We adopt a Conditional Variational Autoencoder to sample meaningful high-dimensional sub-goal candidates and to solve the high-level long-term strategy optimization problem. We evaluate our proposed method in long-horizon driving and robot navigation tasks. Experiments show that our method outperforms baselines with different hierarchical designs and other regular planners without hierarchy in these complex tasks.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 33 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

24 of 33 references.

References · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2017Hindsight Experience Replaycited by 1,136
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  8. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  9. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  11. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  12. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
12 more
  1. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  2. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  3. 2019Goal-conditioned Imitation Learningcited by 162
  4. 2019Planning with Goal-Conditioned Policiescited by 149
  5. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 115
  6. 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 98
  7. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 86
  8. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  9. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  10. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  11. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learningcited by 19
  12. 2021A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 18; 2024: 12; 2025: 11; 2026: 7.

50 papers.

Sort:

By area:

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  2. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  3. 2024Simple Hierarchical Planning with Diffusioncited by 73
  4. 2025Horizon Reduction Makes RL Scalablecited by 57
  5. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 43
  6. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  7. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  8. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  9. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  10. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 15
  11. 2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Controlcited by 14
  12. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstrationcited by 14
38 more
  1. 2023Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skillscited by 14
  2. 2023Efficient Planning with Latent Diffusioncited by 12
  3. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  4. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  5. 2023Hieros: Hierarchical Imagination on Structured State Space Sequence World Modelscited by 8
  6. 2024Diffusion-Reinforcement Learning Hierarchical Motion Planning in Multi-agent Adversarial Gamescited by 7
  7. 2024Adaptive Prediction Ensemble: Improving Out-of-Distribution Generalization of Motion Forecastingcited by 7
  8. 2023Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learningcited by 6
  9. 2023Hybrid Search for Efficient Planning with Completeness Guaranteescited by 6
  10. 2024PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performercited by 6
  11. 2022Safe Reinforcement Learning using Data-Driven Predictive Controlcited by 5
  12. 2025Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planningcited by 5
  13. 2026Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RLcited by 5
  14. 2023Environment Transformer and Policy Optimization for Model-Based Offline Reinforcement Learningcited by 4
  15. 2022BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environmentcited by 3
  16. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioningcited by 3
  17. 2023Goal-conditioned Offline Planning from Curious Explorationcited by 3
  18. 2025Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learningcited by 3
  19. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 3
  20. 2023Backward Learning for Goal-Conditioned Policiescited by 2
  21. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
  22. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learningcited by 2
  23. 2023Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphscited by 1
  24. 2023VAPOR: Legged Robot Navigation in Outdoor Vegetation Using Offline Reinforcement Learningcited by 1
  25. 2024Trajectory Planning for Autonomous Vehicle Using Iterative Reward Prediction in Reinforcement Learningcited by 1
  26. 2025Hierarchical Reinforcement Learning and Value Optimization for Challenging Quadruped Locomotioncited by 1
  27. 2025GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspectivecited by 1
  28. 2025DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learningcited by 1
  29. 2026Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flowscited by 1
  30. 2026SADP: Subgoal-Aware Diffusion Policy for Long-Horizon Manipulation Learned from Foundation Model Generated Demonstrationscited by 1
  31. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
  32. 2023Landmark Guided Active Exploration with State-specific Balance Coefficientcited by 0
  33. 2024Single-Reset Divide & Conquer Imitation Learningcited by 0
  34. 2024Temporal Abstraction in Reinforcement Learning with Offline Datacited by 0
  35. 2024Boosting Hierarchical Reinforcement Learning with Meta-Learning for Complex Task Adaptationcited by 0
  36. 2025Proposing Hierarchical Goal-Conditioned Policy Planning in Multi-Goal Reinforcement Learningcited by 0
  37. 2025Reachability Weighted Offline Goal-conditioned Resamplingcited by 0
  38. 2026SPLC: Social Preference Learning for Crowd Robot Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space
  2. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
  3. 2024Offline Hierarchical Reinforcement Learning via Inverse Optimization
  4. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
  5. 2025Flattening Hierarchies with Policy Bootstrapping
  6. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
  7. 2024Goal-Conditioned Decision Transformer for Multi-Goal Offline Reinforcement Learning
  8. 2026HorizonFlow: Variable-Length Planning for Offline Goal-Conditioned RL