mlsanityData to 7 Oct 2026

ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning

arXiv 2505.22094 ↗First posted 28 May 2025cs.RO cs.LGcited by 83

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We propose ReinFlow, a simple yet effective online reinforcement learning (RL) framework that fine-tunes a family of flow matching policies for continuous robotic control. Derived from rigorous RL theory, ReinFlow injects learnable noise into a flow policy's deterministic path, converting the flow into a discrete-time Markov Process for exact and straightforward likelihood computation. This conversion facilitates exploration and ensures training stability, enabling ReinFlow to fine-tune diverse flow model variants, including Rectified Flow [35] and Shortcut Models [19], particularly at very few or even one denoising step. We benchmark ReinFlow in representative locomotion and manipulation tasks, including long-horizon planning with visual input and sparse reward. The episode reward of Rectified Flow policies obtained an average net growth of 135.36% after fine-tuning in challenging legged locomotion tasks while saving denoising steps and 82.63% of wall time compared to state-of-the-art diffusion RL fine-tuning method DPPO [43]. The success rate of the Shortcut Model policies in state and visual manipulation tasks achieved an average net increase of 40.34% after fine-tuning with ReinFlow at four or even one denoising step, whose performance is comparable to fine-tuned DDIM policies while saving computation time for an average of 23.20%. Project webpage: https://reinflow.github.io/

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 28 of its 49 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

28 of 49 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization9 Feb
  2. 2024Affordance-based Robot Manipulation with Flow Matching2 Sep
  3. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient2 Jun
  4. 2024Riemannian Flow Matching Policy for Robot Motion Learning15 Mar
  5. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  6. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  7. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 20

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2018Neural Ordinary Differential Equationscited by 3,733
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  12. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
8 more
  1. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  2. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  3. 2024One Step Diffusion via Shortcut Modelscited by 373
  4. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  5. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  6. 2019Wasserstein-2 Generative Networkscited by 96
  7. 2023Data Quality in Imitation Learningcited by 72
  8. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Rewardcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 71.

83 papers.

Sort:

By area:

  1. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 65
  2. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  3. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 37
  4. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
  5. 2025One-Step Flow Policy Mirror Descentcited by 18
  6. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  7. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12
  8. 2026VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memorycited by 10
  9. 2026RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulationcited by 10
  10. 2026HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensingcited by 9
  11. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 8
  12. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 7
38 more
  1. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  2. 2026NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Modelscited by 6
  3. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAscited by 5
  4. 2025OMP: One-step Meanflow Policy with Directional Alignmentcited by 4
  5. 2026Reparameterization Flow Policy Optimizationcited by 4
  6. 2026Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Controlcited by 4
  7. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  8. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  9. 2026WCM: A World Critic Model for Vision-Language-Action Reinforcement Learningcited by 4
  10. 2025FlowDrive: moderated flow matching with data balancing for trajectory planningcited by 3
  11. 2025Training Diffusion Policies via Prior-Mapping Co-Evolutioncited by 3
  12. 2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 3
  13. 2026Trust Region Q Adjoint Matchingcited by 3
  14. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policycited by 3
  15. 2025FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learningcited by 2
  16. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learningcited by 2
  17. 2026Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matchingcited by 2
  18. 2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 2
  19. 2026Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worldscited by 2
  20. 2026HiFlow: Tokenization-Free Scale-Wise Autoregressive Policy Learning via Flow Matchingcited by 2
  21. 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Samplingcited by 2
  22. 2026CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Makingcited by 2
  23. 2026RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Rankingcited by 2
  24. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 2
  25. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 2
  26. 2026FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulationcited by 2
  27. 2026Z-1: Efficient Reinforcement Learning for Vision-Language-Action Modelscited by 2
  28. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
  29. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraintscited by 1
  30. 2026ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matchingcited by 1
  31. 2026Mean Flow Policy Optimizationcited by 1
  32. 2026OGPO: Sample Efficient Full-Finetuning of Generative Control Policiescited by 1
  33. 2026Generative Actor-Critic with Soft Bridge Policiescited by 1
  34. 2026Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFTcited by 1
  35. 2026Driving Intents Amplify Planning-Oriented Reinforcement Learningcited by 1
  36. 2026DEFLECT: Temporal Counterfactual Preference Learning for Delay-Robust Asynchronous VLAscited by 1
  37. 2026DriveAnchor: Progressive Anchor-based Flow Learning for Autonomous Driving Planningcited by 1
  38. 2026Set-Supervised Diffusion Policy: Learning Action-Chunking Diffusion through Correctionscited by 1

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation
  2. 2025Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models
  3. 2026Refinement-based Flow Policy Optimization
  4. 2025Iterative Refinement of Flow Policies in Probability Space for Online Reinforcement Learning
  5. 2025Action Flow Matching for Continual Robot Learning
  6. 2026Discrete Flow Matching for Offline-to-Online Reinforcement Learning
  7. 2026Score-Based One-step MeanFlow Policy Optimization
  8. 2026OGPO: One-Step Generative Policy Optimization for Real-Time Robot Control