mlsanityData to 14 Oct 2026

Flow-Based Policy for Online Reinforcement Learning

arXiv 2506.12811 ↗First posted 15 Jun 2025cs.LG cs.AIcited by 37

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

We present FlowRL, a novel framework for online reinforcement learning that integrates flow-based policy representation with Wasserstein-2-regularized optimization. We argue that in addition to training signals, enhancing the expressiveness of the policy class is crucial for the performance gains in RL. Flow-based generative models offer such potential, excelling at capturing complex, multimodal action distributions. However, their direct application in online RL is challenging due to a fundamental objective mismatch: standard flow training optimizes for static data imitation, while RL requires value-based policy optimization through a dynamic buffer, leading to difficult optimization landscapes. FlowRL first models policies via a state-dependent velocity field, generating actions through deterministic ODE integration from noise. We derive a constrained policy search objective that jointly maximizes Q through the flow policy while bounding the Wasserstein-2 distance to a behavior-optimal policy implicitly derived from the replay buffer. This formulation effectively aligns the flow optimization with the RL objective, enabling efficient and value-aware policy learning despite the complexity of the policy class. Empirical evaluations on DMControl and Humanoidbench demonstrate that FlowRL achieves competitive performance in online reinforcement learning benchmarks.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 48 references that are in the map. First posted on arXiv 15 Jun 2025.

Builds on

41 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization9 Feb
  2. 2025Flow Q-Learning4 Feb
  3. 2024Diffusion Policy Policy Optimization1 Sep
  4. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  5. 2024Diffusion Actor-Critic with Entropy Regulator24 May

Other references · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2018Neural Ordinary Differential Equationscited by 3,779
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  12. 2018DeepMind Control Suitecited by 956
24 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  2. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 879
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  6. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  7. 2024Reinforcement learningcited by 247
  8. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  9. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  10. 2023SE(3)-Stochastic Flow Matching for Protein Backbone Generationcited by 150
  11. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 149
  12. 2018Self-Imitation Learningcited by 138
  13. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  14. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 112
  15. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  16. 2024AlphaFold Meets Flow Matching for Generating Protein Ensemblescited by 110
  17. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  18. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  19. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 73
  20. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  21. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  22. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 28
  23. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  24. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 30.

37 papers.

Sort:

By area:

  1. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 71
  2. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 45
  3. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 40
  4. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 31
  5. 2025One-Step Flow Policy Mirror Descentcited by 21
  6. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policiescited by 12
  7. 2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matchingcited by 6
  8. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  9. 2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 5
  10. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  11. 2026Reparameterization Flow Policy Optimizationcited by 4
  12. 2025Training Diffusion Policies via Prior-Mapping Co-Evolutioncited by 3
25 more
  1. 2026Mean Flow Policy Optimizationcited by 3
  2. 2026FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidancecited by 3
  3. 2026Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matchingcited by 2
  4. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 2
  5. 2026What Does Flow Matching Bring To TD Learning?cited by 2
  6. 2026Trust-Region Diffusion Policies for Massively Parallel On-Policy RLcited by 2
  7. 2026DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Modelscited by 2
  8. 2026Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Drivingcited by 1
  9. 2026ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priorscited by 1
  10. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  11. 2026Generative Actor-Critic with Soft Bridge Policiescited by 1
  12. 2026Dual-Flow Reinforcement Learning with State-Aware Explorationcited by 1
  13. 2025FinFlowRL: An Imitation-Reinforcement Learning Framework for Adaptive Stochastic Control in Financecited by 0
  14. 2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Datacited by 0
  15. 2026Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descentcited by 0
  16. 2026Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamicscited by 0
  17. 2026ReFPO: Reflow Regularization for Flow Matching Policy Gradientscited by 0
  18. 2026HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentationcited by 0
  19. 2026Constrained Flow Policy Updates: A Generalized Schr"odinger Bridge Viewcited by 0
  20. 2026Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learningcited by 0
  21. 2026Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policiescited by 0
  22. 2026Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learningcited by 0
  23. 2026Reinforcement Learning via Fixed-point Flow Matchingcited by 0
  24. 2026Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learningcited by 0
  25. 2026Reinforcement Learning via Advantage Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Survey of Flow Matching in Reinforcement Learning
  2. 2026FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification
  3. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
  4. 2026Aligning Flow Map Policies with Optimal Q-Guidance
  5. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies
  6. 2026Reinforcement Learning for Flow-Matching Policies with Density Transport
  7. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
  8. 2025EXPO: Stable Reinforcement Learning with Expressive Policies