mlsanityData to 14 Oct 2026

Prior-Guided Diffusion Planning for Offline Reinforcement Learning

arXiv 2505.10881 ↗First posted 16 May 2025cs.LGcited by 14

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Diffusion models have recently gained prominence in offline reinforcement learning due to their ability to effectively learn high-performing, generalizable policies from static datasets. Diffusion-based planners facilitate long-horizon decision-making by generating high-quality trajectories through iterative denoising, guided by return-maximizing objectives. However, existing guided sampling strategies such as Classifier Guidance, Classifier-Free Guidance, and Monte Carlo Sample Selection either produce suboptimal multi-modal actions, struggle with distributional drift, or incur prohibitive inference-time costs. To address these challenges, we propose Prior Guidance (PG), a novel guided sampling framework that replaces the standard Gaussian prior of a behavior-cloned diffusion model with a learnable distribution, optimized via a behavior-regularized objective. PG directly generates high-value trajectories without costly reward optimization of the diffusion model itself, and eliminates the need to sample multiple candidates at inference for sample selection. We present an efficient training strategy that applies behavior regularization in latent space, and empirically demonstrate that PG outperforms state-of-the-art diffusion policies and planners across diverse long-horizon offline RL benchmarks.Our code is available at https://github.com/ku-dmlab/PG.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 42 of its 50 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

42 of 50 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025What Makes a Good Diffusion Planner for Decision Making?1 Mar
  2. 2025Flow Q-Learning4 Feb
  3. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  4. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  5. 2024Simple Hierarchical Planning with Diffusion5 Jan
  6. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  7. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  8. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
  9. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  10. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  11. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
1 more
  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  9. 2022Video Diffusion Modelscited by 2,305
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  12. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
17 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  4. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  10. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  11. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  12. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  13. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  14. 2021PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Priorcited by 80
  15. 2022Skill-based Model-based Reinforcement Learningcited by 50
  16. 2020Behavior Priors for Efficient Reinforcement Learningcited by 41
  17. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 3
  2. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
  3. 2026Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learningcited by 2
  4. 2026Score-Based One-step MeanFlow Policy Optimizationcited by 2
  5. 2025Actor-Critic without Actorcited by 1
  6. 2026Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learningcited by 1
  7. 2026Direct Soft-Policy Sampling via Langevin Dynamicscited by 1
  8. 2026Improving Diffusion Planners by Self-Supervised Action Gating with Energiescited by 1
  9. 2026Inference Time Policy Optimization for Offline RL with Differentiable World Modelscited by 1
  10. 2026Advantage-Guided Diffusion for Model-Based Reinforcement Learningcited by 1
  11. 2025HOFLON: Hybrid Offline Learning and Online Optimization for Process Start-Up and Grade-Transition Controlcited by 0
  12. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
2 more
  1. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learningcited by 0
  2. 2026HorizonFlow: Variable-Length Planning for Offline Goal-Conditioned RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Modular Diffusion Policy Training: Decoupling and Recombining Guidance and Diffusion for Offline RL
  2. 2024Policy-Guided Diffusion
  3. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RL
  4. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?
  5. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
  6. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  7. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  8. 2026Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies