mlsanityData to 14 Oct 2026

PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer

arXiv 2406.06793 ↗First posted 10 Jun 2024cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Despite the recent advancements in offline RL, no unified algorithm could achieve superior performance across a broad range of tasks. Offline textitvalue function learning, in particular, struggles with sparse-reward, long-horizon tasks due to the difficulty of solving credit assignment and extrapolation errors that accumulates as the horizon of the task grows.~On the other hand, models that can perform well in long-horizon tasks are designed specifically for goal-conditioned tasks, which commonly perform worse than value function learning methods on short-horizon, dense-reward scenarios. To bridge this gap, we propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals. Our experimental results suggest that PlanDQ can achieve superior or competitive performance on D4RL continuous control benchmark tasks as well as AntMaze, Kitchen, and Calvin as long-horizon tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 45 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

37 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Diffused Task-Agnostic Milestone Planner6 Dec
  2. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug
  5. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning24 May

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
20 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  4. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  5. 2021Offline RL Without Off-Policy Evaluationcited by 151
  6. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  7. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 104
  8. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  9. 2022Deep Hierarchical Planning from Pixelscited by 101
  10. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 86
  11. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  12. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  13. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 62
  14. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  15. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 42
  16. 2019Variational Temporal Abstractioncited by 38
  17. 2021Regularized Behavior Value Estimationcited by 32
  18. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
  19. 2023Offline RL with Observation Histories: Analyzing and Improving Sample Complexitycited by 7
  20. 2021Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Monte Carlo Tree Diffusion for System 2 Planningcited by 33
  2. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 13
  3. 2025Fast Monte Carlo Tree Diffusion: 100x Speedup via Parallel Sparse Planningcited by 6
  4. 2026Diffusion ReRoll: Revisable Denoising for Robotic Sequential Predictioncited by 5
  5. 2025Compositional Monte Carlo Tree Diffusion for Extendable Planningcited by 2
  6. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Offline RL with Hierarchical Action Chunking
  2. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  3. 2025Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillation
  4. 2026Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
  5. 2026HorizonFlow: Variable-Length Planning for Offline Goal-Conditioned RL
  6. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  7. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
  8. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning