mlsanityData to 14 Oct 2026

Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning

arXiv 2512.03973 ↗First posted 3 Dec 2025cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline reinforcement learning often relies on behavior regularization that enforces policies to remain close to the dataset distribution. However, such approaches fail to distinguish between high-value and low-value actions in their regularization components. We introduce Guided Flow Policy (GFP), which couples a multi-step flow-matching policy with a distilled one-step actor. The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value actions from the dataset rather than indiscriminately imitating all state-action pairs. In turn, the flow policy constrains the actor to remain aligned with the dataset's best transitions while maximizing the critic. This mutual guidance enables GFP to achieve state-of-the-art performance across 144 state and pixel-based tasks from the OGBench, Minari, and D4RL benchmarks, with substantial gains on suboptimal datasets and challenging tasks. Webpage: https://simple-robotics.github.io/publications/guided-flow-policy/

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 57 references that are in the map. First posted on arXiv 3 Dec 2025.

Builds on

51 of 57 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning20 May
  2. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  3. 2025Flow Q-Learning4 Feb
  4. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  5. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning29 Jul
  6. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
  7. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  8. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning9 Oct
  9. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  10. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  11. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  12. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
4 more
  1. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  3. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling29 Sep
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
23 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  8. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  10. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  11. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 112
  12. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 110
  13. 2023Guided Flows for Generative Modeling and Decision Makingcited by 101
  14. 2024Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulationcited by 93
  15. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  16. 2024Simple Hierarchical Planning with Diffusioncited by 73
  17. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  18. 2025An Introduction to Flow Matching and Diffusion Modelscited by 35
  19. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learningcited by 33
  20. 2024Model-Based Diffusion for Trajectory Optimizationcited by 32
  21. 2023Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Planscited by 27
  22. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  23. 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matchingcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026SVL: Goal-Conditioned Reinforcement Learning as Survival Learningcited by 3
  2. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 2
  3. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 2
  4. 2026Accelerating trajectory optimization with Sobolev-trained diffusion policiescited by 1
  5. 2026Optimal Transport Q-Learning for Flow Policy Steering and Accelerationcited by 0
  6. 2026Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transportcited by 0
  7. 2026G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policiescited by 0
  8. 2026Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policiescited by 0
  9. 2026Value-Guided Repulsion for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VGFM: Expressive Robot Policies via Dense Value Guidance in Flow Matching
  2. 2026Flow Actor-Critic for Offline Reinforcement Learning
  3. 2025Reinforcement Learning for Flow-Matching Policies
  4. 2026Potential-Guided Flow Matching for Vision-Language-Action Policy Improvement
  5. 2026Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning
  6. 2025FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning
  7. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
  8. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning