mlsanityData to 14 Oct 2026

Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows

arXiv 2609.36812 ↗First posted 29 Sep 2026cs.LG cs.AI cs.ROcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 2 more
Abstract (arXiv)

Diffusion and flow policies can model complex behaviors in offline reinforcement learning (RL). However, penalizing their KL divergence from the behavior policy can discourage actions having high critic values with low behavior density. Directly refining behavior proposals may be an alternative, yet Gaussian or deterministic editors limit expressiveness to represent multiple separated modes for the same proposal. In this work, we introduce Proposal-Conditioned Refinement Flows (PReFlow), a policy extraction method combining critic-based proposal selection with a conditional refinement flow. To optimize proposal selection and refinement together, we formulate a KL-regularized objective whose optimum induces a Gibbs policy over final actions under a Gaussian-smoothed behavior prior. The refinement flow can represent multiple high value modes, while a proposal-centered Gaussian reference regulates large action changes. This Gaussian reference further enables us to make use of simulation-free, closed form adjoint matching targets from sampled endpoints and critic gradients, yielding a single velocity regression loss without a backward adjoint solve. On 50 OGBench tasks, PReFlow achieves competitive offline performance and the highest aggregate score among the compared methods after online fine-tuning, reaching 91% after 500K environment steps.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 40 of its 41 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

40 of 41 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching21 Jun
  2. 2026Reversal Q-Learning16 Jun
  3. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policies11 Jun
  4. 2026Efficient Adjoint Matching for Fine-tuning Diffusion Models12 May
  5. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models11 May
  6. 2026Entropy-Regularized Adjoint Matching for Offline Reinforcement Learning7 May
  7. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Map20 Apr
  8. 2026Q-learning with Adjoint Matching20 Jan
  9. 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extraction15 Jan
  10. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning18 Jun
  11. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning20 May
  12. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
8 more
  1. 2025Flow Q-Learning4 Feb
  2. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
  3. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
  4. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  5. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  6. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  7. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 20

  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  7. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  8. 2018Residual Policy Learningcited by 164
  9. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  10. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 115
  11. 2025Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matchingcited by 85
  12. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
8 more
  1. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
  2. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
  3. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  4. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  5. 2026Trust Region Q Adjoint Matchingcited by 4
  6. 2026FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidancecited by 3
  7. 2026Decoupling Policy Extraction for Offline Reinforcement Learningcited by 2
  8. 2026SPAR: Support-Preserving Action Rectificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  2. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning
  3. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation
  4. 2026Counterfactual Transport Flows for Offline Conservative Trajectory Refinement
  5. 2026Refinement-based Flow Policy Optimization
  6. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
  7. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  8. 2026Reinforcement Learning via Fixed-point Flow Matching