mlsanityData to 14 Oct 2026

Diffusion Guidance Is a Controllable Policy Improvement Operator

arXiv 2505.23458 ↗First posted 29 May 2025cs.LGcited by 58

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

also in 1 more
Abstract (arXiv)

At the core of reinforcement learning is the idea of learning beyond the performance in the data. However, scaling such systems has proven notoriously tricky. In contrast, techniques from generative modeling have proven remarkably scalable and are simple to train. In this work, we combine these strengths, by deriving a direct relation between policy improvement and guidance of diffusion models. The resulting framework, CFGRL, is trained with the simplicity of supervised learning, yet can further improve on the policies in the data. On offline RL tasks, we observe a reliable trend -- increased guidance weighting leads to increased performance. Of particular importance, CFGRL can operate without explicitly learning a value function, allowing us to generalize simple supervised methods (e.g., goal-conditioned behavioral cloning) to further prioritize optimality, gaining performance for "free" across the board.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 68 of its 76 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

68 of 76 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Flow Q-Learning4 Feb
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  3. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  4. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control12 Jul
  5. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
  6. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
  7. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  8. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  9. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  10. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning9 Oct
  11. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  12. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
5 more
  1. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  2. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  4. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling29 Sep
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 51

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2022Flow Matching for Generative Modelingcited by 5,702
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  10. 2015Continuous control with deep reinforcement learningcited by 3,407
  11. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  12. 2015Trust Region Policy Optimizationcited by 2,580
38 more
  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2017Hindsight Experience Replaycited by 1,136
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  8. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  13. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  14. 2019Learning Latent Plans from Playcited by 316
  15. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  16. 2024Flow Matching Guide and Codecited by 301
  17. 2024Diffusion Policy Policy Optimizationcited by 280
  18. 2020Critic Regularized Regressioncited by 275
  19. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  20. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  21. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  22. 2021Offline RL Without Off-Policy Evaluationcited by 151
  23. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  24. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  25. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  26. 2021Learning One Representation to Optimize All Rewardscited by 115
  27. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 112
  28. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 110
  29. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  30. 2019Reward-Conditioned Policiescited by 102
  31. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  32. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  33. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
  34. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  35. 2023Anti-Exploration by Random Network Distillationcited by 50
  36. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  37. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 27
  38. 2022Imitating Past Successes can be Very Suboptimalcited by 23

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 48.

58 papers.

Sort:

By area:

  1. 2025π*0.6: a VLA That Learns From Experiencecited by 303
  2. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  3. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 40
  4. 2026Q-learning with Adjoint Matchingcited by 37
  5. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 20
  6. 2025Learn to Guide Your Diffusion Modelcited by 14
  7. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 12
  8. 2026chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistenciescited by 12
  9. 2025Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidancecited by 10
  10. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 10
  11. 2026ViVa: A Video-Generative Value Model for Robot Reinforcement Learningcited by 9
  12. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
38 more
  1. 2025Dichotomous Diffusion Policy Optimizationcited by 6
  2. 2026GlyphPrinter: Region-Grouped Direct Preference Optimization for Glyph-Accurate Visual Text Renderingcited by 6
  3. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  4. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  5. 2025Latent Reasoning in TRMs is Secretly a Policy Improvement Operatorcited by 5
  6. 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learningcited by 5
  7. 2026IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulationcited by 5
  8. 2026Reinforcement Learning via Value Gradient Flowcited by 5
  9. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  10. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policycited by 4
  11. 2026Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learningcited by 3
  12. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 3
  13. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 3
  14. 2026Diffusion Policy through Conditional Proximal Policy Optimizationcited by 2
  15. 2026Latent Policy Steering through One-Step Flow Policiescited by 2
  16. 2026MCLR: Improving Conditional Modeling via Inter-Class Likelihood-Ratio Maximization and Unifying Classifier-Free Guidance with Alignment Objectivescited by 2
  17. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
  18. 2026JEDI: Joint Embedding Diffusion World Model for Online Model-Based Reinforcement Learningcited by 2
  19. 2026Reversal Q-Learningcited by 2
  20. 2026Robot Self-Improvement via Human-Video Dynamics Modelscited by 2
  21. 2025Actor-Critic without Actorcited by 1
  22. 2025Policy Gradient Guidance Enables Test Time Controlcited by 1
  23. 2025ACG: Action Coherence Guidance for Flow-based Vision-Language-Action modelscited by 1
  24. 2026Direct Soft-Policy Sampling via Langevin Dynamicscited by 1
  25. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
  26. 2026FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planningcited by 1
  27. 2026TACO: TActile World Model as a Self-COrrector for Scalable Robot Policy Post-Trainingcited by 1
  28. 2026CLIFT: Turning Gemini Robotics On-Device into Humanoid Specialists via Non-Invasive Closed-Loop Iterative Fine-Tuningcited by 1
  29. 2026PACE: Phase-Progress-Aware Credit for Long-Horizon Embodied Manipulationcited by 1
  30. 2026SmoothRL: Online Reinforcement Learning During Asynchronous Executioncited by 1
  31. 2026Reward Weighted Classifier-Free Guidance as Policy Improvement in Autoregressive Modelscited by 0
  32. 2026ISEP: Implicit Support Expansion for Offline Reinforcement Learning via Stochastic Policy Optimizationcited by 0
  33. 2026Scaling by Diversified Experience for Vision-Language-Action Modelscited by 0
  34. 2026DexPIE: Stable Dexterous Policy Improvement from Real-World Experiencecited by 0
  35. 2026ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learningcited by 0
  36. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  37. 2026ReGuide: From Test-Time Guidance to Self-Improving Diffusion Policiescited by 0
  38. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Guidance Is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Models
  2. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
  3. 2025Modular Diffusion Policy Training: Decoupling and Recombining Guidance and Diffusion for Offline RL
  4. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  5. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning
  6. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies
  7. 2023Diffusion Models for Reinforcement Learning: A Survey
  8. 2025Training Diffusion Policies via Prior-Mapping Co-Evolution