mlsanityData to 14 Oct 2026

TempFlow-GRPO: When Timing Matters for GRPO in Flow Models

arXiv 2508.04324 ↗First posted 6 Aug 2025cs.CVcited by 97

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Recent flow matching models for text-to-image generation have achieved remarkable quality, yet their integration with reinforcement learning for human preference alignment remains suboptimal, hindering fine-grained reward-based optimization. We observe that the key impediment to effective GRPO training of flow models is the temporal uniformity assumption in existing approaches: sparse terminal rewards with uniform credit assignment fail to capture the varying criticality of decisions across generation timesteps, resulting in inefficient exploration and suboptimal convergence. To remedy this shortcoming, we introduce TempFlow-GRPO (Temporal Flow GRPO), a principled GRPO framework that captures and exploits the temporal structure inherent in flow-based generation. TempFlow-GRPO introduces three key innovations: (i) a trajectory branching mechanism that provides process rewards by concentrating stochasticity at designated branching points, enabling precise credit assignment without requiring specialized intermediate reward models; (ii) a noise-aware weighting scheme that modulates policy optimization according to the intrinsic exploration potential of each timestep, prioritizing learning during high-impact early stages while ensuring stable refinement in later phases; and (iii) a seed group strategy that controls for initialization effects to isolate exploration contributions. These innovations endow the model with temporally-aware optimization that respects the underlying generative dynamics, leading to state-of-the-art performance in human preference alignment and text-to-image benchmarks.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 36 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

22 of 36 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  2. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  3. 2025Process Reward Models That Think23 Apr
  4. 2025Process Reinforcement through Implicit Rewards3 Feb
  5. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  6. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec

Other references · 15

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  5. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  6. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  7. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  8. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  9. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  10. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  11. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  12. 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 211
3 more
  1. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  2. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 84
  3. 2024Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimizationcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 74.

97 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  3. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 92
  4. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Modelscited by 67
  5. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 64
  6. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 61
  7. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 49
  8. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 21
  9. 2026Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesiscited by 20
  10. 2025Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Samplingcited by 18
  11. 2025Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learningcited by 18
  12. 2025Flash-DMD: Towards High-Fidelity Few-Step Image Generation with Efficient Distillation and Joint Reinforcement Learningcited by 17
38 more
  1. 2025DiverseGRPO: Mitigating Mode Collapse in Image Generation via Diversity-Aware GRPOcited by 17
  2. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  3. 2026PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Modelscited by 17
  4. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  5. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  6. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
  7. 2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimizationcited by 14
  8. 2026E-GRPO: High Entropy Steps Drive Effective Reinforcement Learning for Flow Modelscited by 14
  9. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  10. 2025Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Modelscited by 12
  11. 2026Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Modelscited by 12
  12. 2025Seeing What Matters: Visual Preference Policy Optimization for Visual Generationcited by 10
  13. 2025SuperFlow: Training Flow Matching Models with RL on the Flycited by 10
  14. 2026TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignmentcited by 10
  15. 2026Stepwise Credit Assignment for GRPO on Flow-Matching Modelscited by 10
  16. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  17. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  18. 2026Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPOcited by 9
  19. 2025Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewardscited by 8
  20. 2026Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Modelscited by 7
  21. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Thinkcited by 7
  22. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 7
  23. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 7
  24. 2025Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignmentcited by 6
  25. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 6
  26. 2026HP-Edit: A Human-Preference Post-Training Framework for Image Editingcited by 6
  27. 2026AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Modelscited by 5
  28. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 5
  29. 2026A Systematic Post-Train Framework for Video Generationcited by 5
  30. 2026Embedding-perturbed Exploration Preference Optimization for Flow Modelscited by 5
  31. 2026HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Modelscited by 4
  32. 2026Faster and Better Alignment for Flow Matching Models via Step-aware Advantagescited by 4
  33. 2026MARBLE: Multi-Aspect Reward Balance for Diffusion RLcited by 4
  34. 2026Efficient Adjoint Matching for Fine-tuning Diffusion Modelscited by 4
  35. 2026Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimizationcited by 4
  36. 2026Alignment of Diffusion Model and Flow Matching for Text-to-Image Generationcited by 3
  37. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Modelscited by 3
  38. 2026Enhancing Spatial Understanding in Image Generation via Reward Modelingcited by 3

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
  2. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Models
  3. 2026LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction
  4. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation
  5. 2026OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models
  6. 2025STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation
  7. 2025Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Models
  8. 2025Flow Matching Policy Gradients