mlsanityData to 7 Oct 2026

Reinforcement Learning for Flow-Matching Policies

arXiv 2507.15073 ↗First posted 20 Jul 2025cs.LGcited by 14

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Flow-matching policies have emerged as a powerful paradigm for generalist robotics. These models are trained to imitate an action chunk, conditioned on sensor observations and textual instructions. Often, training demonstrations are generated by a suboptimal policy, such as a human operator. This work explores training flow-matching policies via reinforcement learning to surpass the original demonstration policy performance. We particularly note minimum-time control as a key application and present a simple scheme for variable-horizon flow-matching planning. We then introduce two families of approaches: a simple Reward-Weighted Flow Matching (RWFM) scheme and a Group Relative Policy Optimization (GRPO) approach with a learned reward surrogate. Our policies are trained on an illustrative suite of simulated unicycle dynamics tasks, and we show that both approaches dramatically improve upon the suboptimal demonstrator performance, with the GRPO approach in particular generally incurring between 50% and 85% less cost than a naive Imitation Learning Flow Matching (ILFM) approach.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 39 of its 40 references that are in the map. First posted on arXiv 20 Jul 2025.

Builds on

39 of 40 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  2. 2025Generative Trajectory Stitching through Diffusion Composition7 Mar
  3. 2025Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization9 Feb
  4. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
  5. 2024Diffusion Policy Policy Optimization1 Sep
  6. 2024Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation6 Mar
  7. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL11 Feb
  8. 2024Simple Hierarchical Planning with Diffusion5 Jan
  9. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  10. 2023Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans30 Oct
  11. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  12. 2023Training Diffusion Models with Reinforcement Learning22 May
1 more
  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 26

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  11. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
14 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  3. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  4. 2025Improving Video Generation with Human Feedbackcited by 267
  5. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  6. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  7. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  8. 2024Vision Language Models are In-Context Value Learnerscited by 84
  9. 2024Sequence-Augmented SE(3)-Flow Matching For Conditional Protein Backbone Generationcited by 36
  10. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  11. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 20
  12. 2025Refined Policy Distillation: From VLA Generalists to RL Expertscited by 19
  13. 2025CHD: Coupled Hierarchical Diffusion for Long-Horizon Taskscited by 6
  14. 2025CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  2. 2025One-Step Flow Policy Mirror Descentcited by 18
  3. 2026HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensingcited by 9
  4. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 8
  5. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 7
  6. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAscited by 5
  7. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5
  8. 2026SA-VLA: Spatially-Aware Flow-Matching for Vision-Language-Action Reinforcement Learningcited by 3
  9. 2026Contrastive Conceptor Activation Steering (COAST): Unlocking Vision-Language-Action Models through Hidden Statescited by 3
  10. 2026Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFTcited by 1
  11. 2026FlowMPC: Improving Flow Matching policies with World Modelscited by 1
  12. 2025RLFR: Extending Reinforcement Learning for LLMs with Flow Environmentcited by 0
2 more
  1. 2026Reinforcement Learning for Flow-Matching Policies with Density Transportcited by 0
  2. 2026DSDyn-VLA: A Dual-Stream Dynamic Manipulation Framework with Motion Perception, Future Awareness, and Realtime Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulation
  2. 2025Extremum Flow Matching for Offline Goal Conditioned Reinforcement Learning
  3. 2025Flow Matching Policy Gradients
  4. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning
  5. 2026ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matching
  6. 2025Dense-Jump Flow Matching with Non-Uniform Time Scheduling for Robotic Policies: Mitigating Multi-Step Inference Degradation
  7. 2025FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning
  8. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning