mlsanityData to 7 Oct 2026

Flow Matching Policy Gradients

arXiv 2507.21053 ↗First posted 28 Jul 2025cs.LG cs.ROcited by 86

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Flow-based generative models, including diffusion models, excel at modeling continuous distributions in high-dimensional spaces. In this work, we introduce Flow Policy Optimization (FPO), a simple on-policy reinforcement learning algorithm that brings flow matching into the policy gradient framework. FPO casts policy optimization as maximizing an advantage-weighted ratio computed from the conditional flow matching loss, in a manner compatible with the popular PPO-clip framework. It sidesteps the need for exact likelihood computation while preserving the generative capabilities of flow-based models. Unlike prior approaches for diffusion-based reinforcement learning that bind training to a specific sampling method, FPO is agnostic to the choice of diffusion or flow integration at both training and inference time. We show that FPO can train diffusion-style policies from scratch in a variety of continuous control tasks. We find that flow-based models can capture multimodal action distributions and achieve higher performance than Gaussian policies, particularly in under-conditioned settings.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 62 of its 77 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

62 of 77 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  2. 2024Diffusion Policy Policy Optimization1 Sep
  3. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  4. 2023Training Diffusion Models with Reinforcement Learning22 May
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov

Other references · 56

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
38 more
  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  8. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  9. 2021Variational Diffusion Modelscited by 1,145
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  12. 2018DeepMind Control Suitecited by 935
  13. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  14. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  15. 2019Solving Rubik's Cube with a Robot Handcited by 699
  16. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  17. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  18. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  19. 2025Mean Flows for One-step Generative Modelingcited by 524
  20. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  21. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  22. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  23. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  24. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  25. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  26. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  27. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 293
  28. 2023Self-Consuming Generative Models Go MADcited by 239
  29. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 198
  30. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  31. 2021A System for General In-Hand Object Re-Orientationcited by 155
  32. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 109
  33. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 101
  34. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  35. 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
  36. 2025MuJoCo Playgroundcited by 57
  37. 2025Visual Imitation Enables Contextual Humanoid Controlcited by 57
  38. 2024Omnigrasp: Grasping Diverse Objects with Simulated Humanoidscited by 45

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 71.

86 papers.

Sort:

By area:

  1. 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 128
  2. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 59
  3. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  4. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  5. 2026Q-learning with Adjoint Matchingcited by 33
  6. 2025Viser: Imperative, Web-based 3D Visualization in Pythoncited by 19
  7. 2025One-Step Flow Policy Mirror Descentcited by 18
  8. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  9. 2026Flow Policy Gradients for Robot Controlcited by 14
  10. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 11
  11. 2026RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulationcited by 10
  12. 2025Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Compositioncited by 8
38 more
  1. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 8
  2. 2026Video Models Can Reason with Verifiable Rewardscited by 8
  3. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 7
  4. 2026ReWorld: Multi-Dimensional Reward Modeling for Embodied World Modelscited by 7
  5. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Thinkcited by 7
  6. 2026Reinforcing Few-step Generators via Reward-Tilted Distribution Matchingcited by 6
  7. 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learningcited by 5
  8. 2026π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAscited by 5
  9. 2026UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptationcited by 5
  10. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 5
  11. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5
  12. 2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 4
  13. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Modelscited by 4
  14. 2026Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Controlcited by 4
  15. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  16. 2025FlowDrive: moderated flow matching with data balancing for trajectory planningcited by 3
  17. 2025Letting Trajectories Spread: Quality-Preserving Control for Diverse Flow Matchingcited by 3
  18. 2025Training Diffusion Policies via Prior-Mapping Co-Evolutioncited by 3
  19. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3
  20. 2025FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learningcited by 2
  21. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learningcited by 2
  22. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
  23. 2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 2
  24. 2026Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worldscited by 2
  25. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 2
  26. 2026FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scalingcited by 2
  27. 2026CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Makingcited by 2
  28. 2026FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidancecited by 2
  29. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 2
  30. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 2
  31. 2026FAIL: Flow Matching Adversarial Imitation Learning for Image Generationcited by 1
  32. 2026MePoly: Max Entropy Polynomial Policy Optimizationcited by 1
  33. 2026Diffusion Policy through Conditional Proximal Policy Optimizationcited by 1
  34. 2026Advances in GRPO for Generation Models: A Surveycited by 1
  35. 2026Quality over Quantity: Demonstration Curation via Influence Functions for Data-Centric Robot Learningcited by 1
  36. 2026ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priorscited by 1
  37. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraintscited by 1
  38. 2026Positive-Only Drifting Policy Optimizationcited by 1

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Refinement-based Flow Policy Optimization
  2. 2026Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models
  3. 2025Reinforcement Learning for Flow-Matching Policies
  4. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  5. 2026Stepwise Credit Assignment for GRPO on Flow-Matching Models
  6. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning
  7. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning
  8. 2026Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Models