mlsanityData to 7 Oct 2026

Flow-GRPO: Training Flow Matching Models via Online RL

arXiv 2505.05470 ↗First posted 8 May 2025cs.CV cs.AIcited by 605

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

We propose Flow-GRPO, the first method to integrate online policy gradient reinforcement learning (RL) into flow matching models. Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation (ODE) into an equivalent Stochastic Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps, enabling statistical sampling for RL exploration; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps, significantly improving sampling efficiency without sacrificing performance. Empirically, Flow-GRPO is effective across multiple text-to-image tasks. For compositional generation, RL-tuned SD3.5-M generates nearly perfect object counts, spatial relations, and fine-grained attributes, increasing GenEval accuracy from 63% to 95%. In visual text rendering, accuracy improves from 59% to 92%, greatly enhancing text generation. Flow-GRPO also achieves substantial gains in human preference alignment. Notably, very little reward hacking occurred, meaning rewards did not increase at the cost of appreciable image quality or diversity degradation.

Key facts. Cited by 605 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 76 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

44 of 76 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 41

  1. 2020Denoising Diffusion Implicit Modelscited by 9,655
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  9. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  10. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  11. 2024OpenAI o1 System Cardcited by 1,220
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 939
29 more
  1. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  2. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  3. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  6. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  7. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  8. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  9. 2025Improving Video Generation with Human Feedbackcited by 267
  10. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  11. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 218
  12. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  13. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  14. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  15. 2025Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distributioncited by 139
  16. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  17. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  18. 2024Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generationcited by 85
  19. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  20. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59
  21. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 57
  22. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  23. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  24. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimizationcited by 38
  25. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 37
  26. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  27. 2024Training-free Diffusion Model Alignment with Sampling Demonscited by 30
  28. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  29. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimizationcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 173; 2026: 432.

605 papers.

Sort:

By area:

  1. 2025Qwen-Image Technical Reportcited by 973
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  3. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 247
  4. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  5. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  6. 2025Seedream 4.0: Toward Next-generation Multimodal Image Generationcited by 233
  7. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  8. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  9. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
  10. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  11. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 168
  12. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 93
38 more
  1. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 88
  2. 2025Flow Matching Policy Gradientscited by 86
  3. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 78
  4. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 76
  5. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 75
  6. 2025LongCat-Video Technical Reportcited by 73
  7. 2025Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Modelcited by 73
  8. 2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Modelscited by 68
  9. 2025RewardDance: Reward Scaling in Visual Generationcited by 63
  10. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 61
  11. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
  12. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 59
  13. 2026Helios: Real Real-Time Long Video Generation Modelcited by 56
  14. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matchingcited by 55
  15. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 50
  16. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 48
  17. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  18. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 43
  19. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 38
  20. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 38
  21. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 35
  22. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  23. 2025HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generationcited by 32
  24. 2025Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preferencecited by 30
  25. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 30
  26. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
  27. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 29
  28. 2026HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformercited by 29
  29. 2025TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Modelscited by 28
  30. 2026Qwen-Image-2.0 Technical Reportcited by 28
  31. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 27
  32. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
  33. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Mapscited by 26
  34. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 25
  35. 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generationcited by 23
  36. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learningcited by 22
  37. 2025OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learningcited by 22
  38. 2026WorldCompass: Reinforcement Learning for Long-Horizon World Modelscited by 22

50 most cited of 605 All 605 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training
  2. 2025Value Gradient Guidance for Flow Matching Alignment
  3. 2026Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
  4. 2026Graph-GRPO: Training Graph Flow Models with Reinforcement Learning
  5. 2026MEND: RL For Flow Models via Proximal Velocity Matching
  6. 2025Fine-tuning Flow Matching Generative Models with Intermediate Feedback
  7. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
  8. 2026Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning